Sunday 17 February 2019

Getting a Linux container to connect to the Cosmos DB emulator on the host machine

I've finally got a Linux container to connect to the Cosmos DB emulator on the host machine.

It is necessary to override the SSL verification process because the cURL /OpenSSL implementation on Debian Linux will return an error that the emulator is returning a self-signed certificate. This happens even if you install the certificate into ca-certificates.

This doesn't happen on Windows containers; it will honour the certificate if you install it into the Trusted Certification Authorities on the Computer store.

The other thing I found with using the V3 Cosmos DB SDK is that it is necessary to specifically set the connection mode to ConnectionMode = ConnectionMode.Gateway. If you don't do this you get the error:

CosmosRequestException;StatusCode=ServiceUnavailable;SubStatusCode=0;ActivityId=4a705616-21ae-4d5d-bec5-7b9ee8950438;RequestCharge=0;Message=Response status code does not indicate success: 503 Substatus: 0 Reason: (Service is currently unavailable.ActivityId: 4a705616-21ae-4d5d-bec5-7b9ee8950438, RequestStartTime: 2019-02-17T19:22:47.4337274Z, Number of regions attempted: 5ResponseTime: 2019-02-17T19:22:47.7375857Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.7382707Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.7514215Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.7514246Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.7593737Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.7593837Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.7670388Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.7670421Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.7757066Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.7757105Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.7824268Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8019489Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8019536Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8103823Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8103858Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8184272Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8184305Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8287382Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8287428Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8385530Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8385565Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8465729Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8750780Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8926065Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9215927Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9215972Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9327865Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9327900Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9408781Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9408819Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9484237Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9484273Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9575228Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9575270Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9652242Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9768492Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9925856Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.0751316Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.0751477Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.1198555Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.1198862Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.1598310Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.1598465Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.1979447Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.1979610Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.2444786Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.2445010Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.2861704Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.3307031Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.4127985Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.8942345Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.8942415Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.9085654Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.9085690Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.9239002Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.9239039Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.9358565Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.9358599Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.9481837Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.9481912Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.9594525Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.9686429Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.9907359Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.2019523Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.2019720Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.2827287Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.2827470Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.3335009Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.3335123Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.3591186Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.3591293Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.3821192Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.3821303Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.4111658Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.4410806Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.5026776Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.7159972Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.7160032Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.7437652Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.7437736Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.7613897Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.7613939Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.7719120Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.7719163Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.7846943Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.7846984Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8001696Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8108664Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8336267Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.1608398Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.1608511Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.1960636Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.1960839Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.2297966Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.2298226Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.2711731Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.2711910Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.3021908Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.3022073Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.3437017Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.3568721Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.3809056Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.5724733Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.5724823Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.5942277Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.5942357Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.6160218Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.6160287Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.6295680Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.6295724Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.6511830Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.6511884Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.6610954Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.6736195Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.6948315Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.0106273Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.0106327Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.0180594Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.0180628Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.0319283Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.0319320Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.0432484Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.0432519Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.0597979Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.0598037Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.0752649Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.0916680Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.1067702Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.5288492Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.5288572Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.5425668Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.5425706Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.5533611Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.5533685Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.5644832Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.5644870Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.5879979Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.5880046Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.6040075Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.6195609Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.6461146Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8490081Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8490126Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8563830Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8563870Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8652515Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8652560Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8743458Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8743500Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8830416Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8830475Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8920612Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.9006524Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.9167751Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadAddressResolution - StartTime: 2019-02-17T19:22:47.5037673Z, EndTime: 2019-02-17T19:22:47.5622606Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:47.7836648Z, EndTime: 2019-02-17T19:22:47.7917256Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:47.8759944Z, EndTime: 2019-02-17T19:22:47.8827398Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:47.9058245Z, EndTime: 2019-02-17T19:22:47.9116592Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:47.9769286Z, EndTime: 2019-02-17T19:22:47.9833511Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:48.9882031Z, EndTime: 2019-02-17T19:22:49.0234257Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:49.3309670Z, EndTime: 2019-02-17T19:22:49.3652992Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:49.4514450Z, EndTime: 2019-02-17T19:22:49.5389936Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:49.6737033Z, EndTime: 2019-02-17T19:22:49.6823786Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:51.8055199Z, EndTime: 2019-02-17T19:22:51.8676713Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:51.9687320Z, EndTime: 2019-02-17T19:22:51.9759642Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:51.9913654Z, EndTime: 2019-02-17T19:22:51.9978117Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:52.0918485Z, EndTime: 2019-02-17T19:22:52.0979730Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:56.1052731Z, EndTime: 2019-02-17T19:22:56.1462273Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:56.4412705Z, EndTime: 2019-02-17T19:22:56.4799560Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:56.5030653Z, EndTime: 2019-02-17T19:22:56.5140179Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:56.6196482Z, EndTime: 2019-02-17T19:22:56.6319983Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:23:04.6485699Z, EndTime: 2019-02-17T19:23:04.6882551Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:23:04.8109491Z, EndTime: 2019-02-17T19:23:04.8228116Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:23:04.8339162Z, EndTime: 2019-02-17T19:23:04.8408803Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:23:04.9007391Z, EndTime: 2019-02-17T19:23:04.9069765Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:23:17.9146970Z, EndTime: 2019-02-17T19:23:18.1311934Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:23:18.3569803Z, EndTime: 2019-02-17T19:23:18.3656629Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbd, Linux/9 documentdb-netcore-sdk/2.0.0).;

So you must connect with:
var cosmosConfiguration = new CosmosConfiguration(accountEndpoint, accountKey)
                {
                    DisableSslVerification = true,
                    ConnectionMode = ConnectionMode.Gateway
                };
Posted by at No comments:

Friday 15 February 2019

Linux SSL Certificate verification in .NET Core

This is some analysis into the verification of SSL in .NET Core.
There is a behavioural difference: it you add a self-signed certificate into "Trusted Publishers" on a Windows machine then self-signed certificates are accepted, but on Linux machines they still error.

System.Net.Security.SecureChannel:VerifyRemoteCertificate

If CompleteHandshake fails then the error is thrown:
SR.net_ssl_io_cert_validation  "The remote certificate is invalid according to the validation procedure."

CompleteHandshake calls VerifyRemoteCertificate

SecureChannel.VerifyRemoteCertificate returns false if the certificate cannot be validated.

It calls CertificateValidationPal.GetRemoteCertificate and then CertificateValidationPal....

At this point the logic forks depending upon the OS.
CertificateValidationPal.VerifyCertificateProperties is in CertificateValidationPal.Unix.cs.

For Unix, this calls the CertificateValidation.BuildChainAndVerifyProperties.
This is in CertificateValidation.Unix.cs.

The X509Chain is built. It returns a ChainPal and runs BuildChain on it.
A linux ChainPal is returned.
It then runs the OpenSslX509ChainProcessor and runs BuildChain.
This has recursion as it enumerates the chain. It has a timeout to ensure the loading of certificates does not take too long.

The OpenSslX509ChainProcessor is run, and it runs the Pal.Unix/ChainVerifier.cs.
This runs HasUnsuppressedError which checks for errors that have not been surpressed. This includes checking for self-signed certificates.

The  Interop.Crypto.X509VerifyCert is called which calls calls native C code which calls the openSSL library function X509_verify_cert. This calls verify_chain.
OpenSSL build_chain will check for self signed certificates which has a return value of 18.

In OpenSSL, a self-signed certificate has a chain of 1.
Running 
openssl s_client -showcerts -verify 5 -connect cosmosdb.test:8081 
against a server will retun 

Debian ca-certificates notes.

Verify return code: 18 (self signed certificate)

In the dotnet core implhementation for Unix, Self-signed is when the subject name is the same as the issuer name (see code). Errors can occur because the peer trust is only supported for Windows certificate stores. Some of this is discussed here.

Having a self-signed certificate raises the X509VerificationFlags.IgnoreRootRevocationUnknown flag which needs to be surpressed, otherwise an error is returned.


This is a subtle difference from the standard Linux implementation of dotnet core; in dotnet core Self-signed is when the subject name is the same as the issuer name (see code).

If you have running Windows and a self-signed certificate in the Trusted Publishers store, then dotnet core will respect it and not throw an error. If you are running the same code on Linux and have installed the self-signed certificate into the ca-certificates store, it will throw an error.

The X509 Store differs by platform. 
The LocalMachine\Root store on Linux is an interpretation of the CA bundle in the default path for OpenSSL. The LocalMachine\Intermediate store on Linux is an interpretation of the CA bundle in the default path for OpenSSL. The CurrentUser\Intermediate store on Linux is used as a cache when downloading intermediate CAs by their Authority Information Access records on successful X509Chain builds.





Posted by at No comments:

Sunday 10 February 2019

Docker commands & shortcuts

To stop all docker containers, in PowerShell:

docker ps -a -q | ForEach { docker stop $_ }
Posted by at No comments:

Using the Cosmos DB emulator across the network

It is very useful to be able to use the Cosmos DB emulator across the network; a good example being that you may be building Linux docker images to run on Kubernetes and you want them to access the Cosmos DB Emulator running on your development Windows desktop. Currently the Cosmos DB emulator only runs inside a Windows container, so running it as a container here won't help.

This article describes some of the steps to get you there.

I start off with installing the emulator on a development PC (the server) and getting a different Windows machine to access the emulator across the network. This proves the connectivity and then I move to configuring my Linux docker app.

 CLIENT: Configure a hostname for accessing the emulator on the client PC
I am using the hostname cosmosdb.test to access the emulator.

In the client machine, running as an administrator edit the hosts file C:\windows\system32\drivers\etc\hosts and add the line:

<my emulator machine ip> cosmosdb.test

Note that the <my emulator machine ip> cannot be 127.0.0.1 as any running container will resolve this locally and try and connect to itself rather than the host machine. It has to be the host machines real IP address! (Another alternative would be to use the host.docker.internal hostname throughout).

Flush the local DNS to be sure (especially if you've run this step before)
ipconfig /flushdns

SERVER: Install and Run the emulator
Install the Cosmos DB Emulator.

Start it.

Stop it.

SERVER: Configure Windows Defender Firewall
Ensure Cosmos DB Emulator has a firewall exception for port 8081.

Start > Run > Windows Defender Firewall > Advanced Settings > Inbound Rules > New Rule

Select Port > TCP and Specific Local Ports > Enter 8081 > Allow the Connection

Select the networks to which the rule should apply. If you are really unsure, select Domain, Network & Public (but at your own risk!)

Give the rule and name and description and select OK.

SERVER: Generate a self-signed SSL certificate allowing the host name

Start the emulator from the command-line, adding the parameter
/GenCert=cosmosdb.test,<IP address>

Make a note of the certificate thumbprint (the last 5 digits may be sufficient).

SERVER: Export the SSL certificate
Start > Manage Computer Certificates

Right click on the root node and select Find Certificates

Change Look in Field to SHA1 hash and enter the last 5 digits of the thumbprint from the previous step into the Contains field.

You should see one (or more) entries for the Cosmos DB certificate.

Right-click on one and select Export.

Select No, do not export the private key.

Select Base-64 encoded X.509 (CER)

Save to a file. We will save it as cosmosdbemulator.cer (and we'll refer to that filename later).

CLIENT: Import the SSL certificate on the client machine
On the client machine:

Start > Manage Computer Certificates

Select Trusted Root Certification Authorities > Certificates

 Select All Tasks > Import > Next

Select the filename and click Next until the certificate is imported.

Double-click on the newly imported certificate, select the Details tab and confirm that the Subject Alternative Name property includes cosmosdb.test.

 SERVER: Create a new access key

 Start the emulator from the command-line, adding the parameter
/GenKeyFile=MyKeyFile.txt

Open the key file and make a note of the access key as this will be set explicitly when we start the emulator in future.

In my example I will use:
1rAUqgeKitqNhTKyKMtFQ7UK79d9cQhEfoiXomlh3zc1Qz58uQQV6c49B2wGeC9FuGM+OmRViCHRuJu3llb0Vg==

 SERVER: Configure and start the emulator

The next steps configure the emulator for allowing access across the network. You may want to modify the emulator shortcut to set these command-line parameters, or run the executable directly from the command-line. Just remember, in future you must always start the emulator with certain additional parameters.

Start the emulator with the following parameters:
/NoFirewall /AllowNetworkAccess /Key=<access key>

or in my example
/NoFirewall /AllowNetworkAccess /Key=1rAUqgeKitqNhTKyKMtFQ7UK79d9cQhEfoiXomlh3zc1Qz58uQQV6c49B2wGeC9FuGM+OmRViCHRuJu3llb0Vg==

The AllowNetworkAccess must be set for the emulator to accept connections across the network.

CLIENT: Check connectivity
On the client PC open the command prompt and type
telnet cosmosdb.test 8081

And you should connect. If you don't then your host name is pointing to the wrong IP address or the firewall is still blocking you. If so, check the previous steps.

CLIENT: Run the test code
Run the test app on the client PC.

I have found that the V3 Beta Cosmos DB SDK works better when connecting across the network. I recommend you use this for all new development.

using System;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Azure.Cosmos;
using Newtonsoft.Json;

namespace cosmosdb_test_v3
{
    class Program
    {
        static void Main(string[] args)
        {
            var cts = new CancellationTokenSource();

            Console.CancelKeyPress += (s, e) =>
            {
                e.Cancel = true;
                cts.Cancel();
            };

            MainAsync(args, cts.Token).Wait(cts.Token);

            Console.WriteLine("Program exited");
        }

        private static async Task MainAsync(string[] args, CancellationToken token)
        {
            try
            {
                const string databaseId = "testDb";
                const string accountEndpoint = "https://cosmosdb.test:8081";
                const string accountKey = "1rAUqgeKitqNhTKyKMtFQ7UK79d9cQhEfoiXomlh3zc1Qz58uQQV6c49B2wGeC9FuGM+OmRViCHRuJu3llb0Vg==";
                const string containerId = "testCollection";
                const string partitionKeyPath = "/id";
                var documentId = "1";

                if (args.Length > 0)
                {
                    documentId = args[0];
                }

                var cosmosConfiguration = new CosmosConfiguration(accountEndpoint, accountKey);
                using (var client = new CosmosClient(cosmosConfiguration))
                {
                    CosmosDatabase database = await client.Databases.CreateDatabaseIfNotExistsAsync(databaseId, cancellationToken: token);
                    CosmosContainer container = await database.Containers.CreateContainerIfNotExistsAsync(containerId, partitionKeyPath, cancellationToken: token);

                    var person = new Person
                    {
                        Id = "Bloggs_" + documentId,
                        FirstName = "Joe",
                        LastName = "Bloggs_" + documentId,
                        Age = 35
                    };

                    await container.Items.CreateItemAsync(person.Id, person, cancellationToken: token);
                    Console.WriteLine("Document " + documentId + " created");
                }
            }
            catch (Exception e)
            {
                Console.WriteLine(e);
            }
        }
    }

    internal class Person
    {
        [JsonProperty("id")]
        public string Id { get; set; }
        public string FirstName { get; set; }
        public string LastName { get; set; }
        public int Age { get; set; }
    }
}

SERVER: Confirm that the document has been written
Open Data Explorer and confirm that the document has been written to the emulator.

Congratulations! You should now have run the dotnet core console application on one machine and connected to your development machine. Now we are going to run the same dotnet core console app as a Linux container.

Run Client Code in a Linux Docker container
Create a cert folder in the project folder and copy cosmosdbemulator.cer into that folder.
Rename the certificate to use a crt extension: cosmosdbemulator.crt.

Add the following Dockerfile to the root of the project:

FROM microsoft/dotnet:2.1-sdk
WORKDIR /app

# copy csproj and restore as distinct layers
COPY *.csproj ./
RUN dotnet restore


# copy the SSL certificate
ADD cert/* /tmp/cert/
RUN apt-get install ca-certificates && \
    cp -R /tmp/cert/cosmosdbemulator.crt /usr/local/share/ca-certificates && \
    update-ca-certificates

# copy and build everything else
COPY . ./
RUN dotnet publish -c Release -o out
ENTRYPOINT ["dotnet", "out/cosmosdb-test-v3.dll"]




Using the command-line in the root of the project, type



docker build -t cosmosdb-test-v3 .



Look for the text

Updating certificates in /etc/ssl/certs...

1 added, 0 removed; done.



to check that the SSL certificate has been installed correctly.



Note for update-ca-certificates to work the certificate should be found in the /usr/local/share/ca-certificates folder and have the .crt file extension. A different file extension (such as .cer) will not work.



Now run the test app container.



docker run --rm cosmosdb-test-v3 <my document number>



You can set any document number (just don't repeat a number or you will get a duplicate document exception!)



docker run --rm cosmosdb-test-v3 15



You should see

Document 15 created

Program exited



Congratulations! You now have a Linux docker container running the dotenet core Cosmos DB V3 SDK, writing a document to your host development emulator.



Debugging the certificates

If at any time you need to check the successful installation of the certificates on the Linux container you can run the container with a shell:

docker run --entrypoint /bin/bash -it cosmosdb-test-v3



You can check connectivity with:

curl -kv https://host.docker.internal:8081 and

curl -kv https://cosmosdb.test:8081



You can check the specific certificate would work if it were installed in the CA store using:

curl -v https://cosmosdb.test:8081 --cacert /tmp/cert/<mycert>.crt



Subsequent Notes

This works only if you connect your Cosmos DB client to another machine on the network; it doesn't work if you connect your Cosmos DB client to the same machine.











Posted by



at





No comments:
              
































        

      









Subscribe to:
Posts (Atom)