Tuesday, 19 November 2019

Chaos testing Azure Service Bus with Toxiproxy

Toxiproxy

Toxiproxy runs as a docker container.

On Docker for Windows, run it in bridge networking (the default). In this the containers are isolated and so to access the port from the host it must be published. Run the container with:
docker run -it -p 8474:8474 -p 443:443 -p 5671:5671 -p 5672:5672 shopify/toxiproxy

This exposes port 8474 (the HTTP API), 443 for HTTPS and ports 5671 and 5672 for proxying Azure Service Bus on AMQP.

You can configure the proxy either by using the client SDK or by using the shell.
To attach the shell:
Docker exec -it <container ID> /bin/sh


To create a proxy for AMQP for ASB, first determine the IP address of the service.
NSLlookup <FQDN of ASB service>


In the shell type:
./toxiproxy-cli create amqp --listen 0.0.0.0:5671 --upstream <IP address of ASB service>:5671


To add upstream and downstream latency:
./toxiproxy-cli toxic add -n downlatency -t latency -a latency=8000 -d amqp

./toxiproxy-cli toxic add -n uplatency -t latency -a latency=8000 -u amqp

Tips

If you need to test any network connectivity you can use
docker run --rm -it praqma/network-multitool bash

Friday, 16 August 2019

Correlation in Application Insights

operation_parentId does not hold the value of the parent's operation_id. Instead it holds the value of the parent's request id. Confusing, eh?

operation_id has to be set in every logical service.

the request id has to be globally unique for every call, so you cannot set it to be a business key, such as an order reference.


Tuesday, 13 August 2019

Docker for Windows Desktop clock skew: Cosmos DB "The authorization token is not valid at the current time. Please create another token and retry"

I had a containerised application that connects to my Cosmos DB emulator running on my development environment.

Sometimes when I went back to the app after a couple of days it would fail. The Cosmos DB client would error with:

"The authorization token is not valid at the current time. Please create another token and retry"

Looking at the times in the message they were earlier than the current time so that token was out of date.

Attaching to the container with:

docker exec -it <container ID> /bin/bash

I could see that the container time was indeed wrong.

Solution 1

The solution was to restart Docker for Windows desktop.

Solution 2

The other solution is to go into Hyper-V Manager, select the Linux VM, select the settings, Integration Services and then disable and then re-enable Time Synchronisation.

Saturday, 15 June 2019

View HikVision cameras in VLC player

The camera feed is located at:

rtsp://<ipaddress>:554/ch1/main/av_stream

Create the following registry file and execute it:
Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\rtsp]
@="Real Time Streaming Protocol"

"URL Protocol"=""
[HKEY_CLASSES_ROOT\rtsp\shell]
[HKEY_CLASSES_ROOT\rtsp\shell\open]
[HKEY_CLASSES_ROOT\rtsp\shell\open\command]
@="\"C:\\Program Files\\VideoLan\\VLC\\vlc.exe\"%1"

Thursday, 13 June 2019

Android 9 breaks audio volume sync on JVC HA-S30BT bluetooth headphones

On Android 8 the media volume controls adjust the volume played back by the headset.
After applying the upgrade to Android 9 the controls seemed sporadic: sometimes the handset volume controls would adjust the headphones, sometimes it would not.

If you go into the Settings > Connections > Bluetooth > ... > Advanced the "Media Volume Sync" setting was greyed out and it said underneath "Not supported by the connected Bluetooth device".

This post seemed to fix it. By renaming the headphones to ABS-DO-ENABLE and disconnecting and reconnecting the handset volume then controlled the bluetooth headphone volume.

Once you've got it working, renaming it back seems to continue to work.

Monday, 8 April 2019

Recording the audio from the computer

I had to archive the audio from a radio stream.

For this you can use Audacity, use the Windows Audio Session API (WASAPI) and select the speaker as the loopback.

https://www.howtogeek.com/217348/how-to-record-the-sound-coming-from-your-pc-even-without-stereo-mix/

Sunday, 10 March 2019

Cosmos DB emulator returns wrong IP address

I configured the Cosmos DB emulator to listen across the network. It was running on a developer laptop that had Docker for Windows installed.

Connecting from a different Linux laptop using Gateway mode, I was getting long delays and then "a task was cancelled". This was coming from deep within the HTTP stack.

On closer inspection I could see that the client was trying to connect to https://10.0.75.1 - this is the Docker network.

In the GetDatabaseAccountAsync method the client connects to the gateway and enumerates the databases:

https://cosmosdb.test:8081/

The gateway returns a JSON response. This can be seen in Visual Studio Code with the following debug query:

System.Text.Encoding.ASCII.GetString(((System.IO.MemoryStream)documentServiceResponse.ResponseBody).ToArray())

The databaseAccountEndpoint property returns https://10.0.75.1:8081 and this is used to form subsequent responses.
{
    "_self": "",
    "id": "localhost",
    "_rid": "cosmosdb.test",
    "media": "//media/",
    "addresses": "//addresses/",
    "_dbs": "//dbs/",
    "writableLocations": [
        {
            "name": "South Central US",
            "databaseAccountEndpoint": "https://10.0.75.1:8081/"
        }
    ],
    "readableLocations": [
        {
            "name": "South Central US",
            "databaseAccountEndpoint": "https://10.0.75.1:8081/"
        }
    ],
    "enableMultipleWriteLocations": false,
    "userReplicationPolicy": {
        "asyncReplication": false,
        "minReplicaSetSize": 1,
        "maxReplicasetSize": 4
    },
    "userConsistencyPolicy": {
        "defaultConsistencyLevel": "Session"
    },
    "systemReplicationPolicy": {
        "minReplicaSetSize": 1,
        "maxReplicasetSize": 4
    },
    "readPolicy": {
        "primaryReadCoefficient": 1,
        "secondaryReadCoefficient": 1
    },
    "queryEngineConfiguration": "{\\"maxSqlQueryInputLength\\":262144,\\"maxJoinsPerSqlQuery\\":5,\\"maxLogicalAndPerSqlQuery\\":500,\\"maxLogicalOrPerSqlQuery\\":500,\\"maxUdfRefPerSqlQuery\\":10,\\"maxInExpressionItemsCount\\":16000,\\"queryMaxInMemorySortDocumentCount\\":500,\\"maxQueryRequestTimeoutFraction\\":0.9,\\"sqlAllowNonFiniteNumbers\\":false,\\"sqlAllowAggregateFunctions\\":true,\\"sqlAllowSubQuery\\":true,\\"sqlAllowScalarSubQuery\\":true,\\"allowNewKeywords\\":true,\\"sqlAllowLike\\":false,\\"sqlAllowGroupByClause\\":false,\\"maxSpatialQueryCells\\":12,\\"spatialMaxGeometryPointCount\\":256,\\"sqlAllowTop\\":true,\\"enableSpatialIndexing\\":true}"
}

TomTom 6100 has a 32GB SD card limit

The TomTom 6100 has a 32GB maximum SD card limit because it requires the FAT32 file system.

Friday, 8 March 2019

Clone multiple-partition SD card with dd

List the disks and partitions
fdisk -l

Assuming that the sdcard is /dev/sdb with partitions /dev/sdb1 and /dev/sdb2

Unmount the partitions
umount /dev/sdb1
umount /dev/sdb2

Backup the partition table & MBR
dd if=/dev/sdb of=partitiontable.img bs=512 count=1

Backup the partitions
dd if=/dev/sdb1 of=partition1.img
dd if=/dev/sdb2 of=partition2.img

Restore the partition & MBR

Restore the partitions
sudo dd if=partitiontable.img of=/dev/sdb
sudo dd if=partition1.img of=/dev/sdb1
sudo dd if=partition2.img of=/dev/sdb2

https://askubuntu.com/questions/491082/steps-to-create-dd-image-file-from-usb-and-restore-image-to-a-different-usb




Attempt 2
umount /dev/sdb1
umount /dev/sdb2
dd if=/dev/sdb of=sdb.img bs=4M conv=noerror,notrunc status=progress

Sunday, 17 February 2019

Getting a Linux container to connect to the Cosmos DB emulator on the host machine

I've finally got a Linux container to connect to the Cosmos DB emulator on the host machine.

It is necessary to override the SSL verification process because the cURL /OpenSSL implementation on Debian Linux will return an error that the emulator is returning a self-signed certificate. This happens even if you install the certificate into ca-certificates.

This doesn't happen on Windows containers; it will honour the certificate if you install it into the Trusted Certification Authorities on the Computer store.

The other thing I found with using the V3 Cosmos DB SDK is that it is necessary to specifically set the connection mode to ConnectionMode = ConnectionMode.Gateway. If you don't do this you get the error:

CosmosRequestException;StatusCode=ServiceUnavailable;SubStatusCode=0;ActivityId=4a705616-21ae-4d5d-bec5-7b9ee8950438;RequestCharge=0;Message=Response status code does not indicate success: 503 Substatus: 0 Reason: (Service is currently unavailable.ActivityId: 4a705616-21ae-4d5d-bec5-7b9ee8950438, RequestStartTime: 2019-02-17T19:22:47.4337274Z, Number of regions attempted: 5ResponseTime: 2019-02-17T19:22:47.7375857Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.7382707Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.7514215Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.7514246Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.7593737Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.7593837Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.7670388Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.7670421Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.7757066Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.7757105Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.7824268Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8019489Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8019536Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8103823Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8103858Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8184272Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8184305Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8287382Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8287428Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8385530Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8385565Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8465729Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8750780Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.8926065Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9215927Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9215972Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9327865Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9327900Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9408781Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9408819Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9484237Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9484273Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9575228Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9575270Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9652242Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9768492Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:47.9925856Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.0751316Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.0751477Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.1198555Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.1198862Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.1598310Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.1598465Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.1979447Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.1979610Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.2444786Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.2445010Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.2861704Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.3307031Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.4127985Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.8942345Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.8942415Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.9085654Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.9085690Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.9239002Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.9239039Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.9358565Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.9358599Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.9481837Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.9481912Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.9594525Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.9686429Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:51.9907359Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.2019523Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.2019720Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.2827287Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.2827470Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.3335009Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.3335123Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.3591186Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.3591293Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.3821192Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.3821303Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.4111658Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.4410806Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.5026776Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.7159972Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.7160032Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.7437652Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.7437736Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.7613897Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.7613939Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.7719120Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.7719163Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.7846943Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.7846984Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8001696Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8108664Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8336267Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.1608398Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.1608511Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.1960636Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.1960839Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.2297966Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.2298226Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.2711731Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.2711910Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.3021908Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.3022073Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.3437017Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.3568721Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:18.3809056Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.5724733Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.5724823Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.5942277Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.5942357Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.6160218Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.6160287Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.6295680Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.6295724Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.6511830Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.6511884Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.6610954Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.6736195Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:49.6948315Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.0106273Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.0106327Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.0180594Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.0180628Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.0319283Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.0319320Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.0432484Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.0432519Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.0597979Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.0598037Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.0752649Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.0916680Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:52.1067702Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.5288492Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.5288572Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.5425668Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.5425706Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.5533611Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.5533685Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.5644832Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.5644870Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.5879979Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.5880046Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.6040075Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.6195609Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:22:56.6461146Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8490081Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8490126Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8563830Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8563870Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8652515Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8652560Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8743458Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8743500Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8830416Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8830475Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.8920612Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.9006524Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadResponseTime: 2019-02-17T19:23:04.9167751Z, StoreReadResult: StorePhysicalAddress: rntbd://172.21.108.33:10251/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 410, IsGone: True, IsNotFound: False, IsInvalidPartition: False, RequestCharge: 0, ItemLSN: -1, SessionToken: , ResourceType: Database, OperationType: ReadAddressResolution - StartTime: 2019-02-17T19:22:47.5037673Z, EndTime: 2019-02-17T19:22:47.5622606Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:47.7836648Z, EndTime: 2019-02-17T19:22:47.7917256Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:47.8759944Z, EndTime: 2019-02-17T19:22:47.8827398Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:47.9058245Z, EndTime: 2019-02-17T19:22:47.9116592Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:47.9769286Z, EndTime: 2019-02-17T19:22:47.9833511Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:48.9882031Z, EndTime: 2019-02-17T19:22:49.0234257Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:49.3309670Z, EndTime: 2019-02-17T19:22:49.3652992Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:49.4514450Z, EndTime: 2019-02-17T19:22:49.5389936Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:49.6737033Z, EndTime: 2019-02-17T19:22:49.6823786Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:51.8055199Z, EndTime: 2019-02-17T19:22:51.8676713Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:51.9687320Z, EndTime: 2019-02-17T19:22:51.9759642Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:51.9913654Z, EndTime: 2019-02-17T19:22:51.9978117Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:52.0918485Z, EndTime: 2019-02-17T19:22:52.0979730Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:56.1052731Z, EndTime: 2019-02-17T19:22:56.1462273Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:56.4412705Z, EndTime: 2019-02-17T19:22:56.4799560Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:56.5030653Z, EndTime: 2019-02-17T19:22:56.5140179Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:22:56.6196482Z, EndTime: 2019-02-17T19:22:56.6319983Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:23:04.6485699Z, EndTime: 2019-02-17T19:23:04.6882551Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:23:04.8109491Z, EndTime: 2019-02-17T19:23:04.8228116Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:23:04.8339162Z, EndTime: 2019-02-17T19:23:04.8408803Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:23:04.9007391Z, EndTime: 2019-02-17T19:23:04.9069765Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:23:17.9146970Z, EndTime: 2019-02-17T19:23:18.1311934Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbdAddressResolution - StartTime: 2019-02-17T19:23:18.3569803Z, EndTime: 2019-02-17T19:23:18.3656629Z, TargetEndpoint: https://172.21.108.33:8081//addresses/?$resolveFor=dbs&$filter=protocol eq rntbd, Linux/9 documentdb-netcore-sdk/2.0.0).;

So you must connect with:
var cosmosConfiguration = new CosmosConfiguration(accountEndpoint, accountKey)
                {
                    DisableSslVerification = true,
                    ConnectionMode = ConnectionMode.Gateway
                };

Friday, 15 February 2019

Linux SSL Certificate verification in .NET Core

This is some analysis into the verification of SSL in .NET Core.
There is a behavioural difference: it you add a self-signed certificate into "Trusted Publishers" on a Windows machine then self-signed certificates are accepted, but on Linux machines they still error.

System.Net.Security.SecureChannel:VerifyRemoteCertificate

If CompleteHandshake fails then the error is thrown:
SR.net_ssl_io_cert_validation  "The remote certificate is invalid according to the validation procedure."

CompleteHandshake calls VerifyRemoteCertificate

SecureChannel.VerifyRemoteCertificate returns false if the certificate cannot be validated.

It calls CertificateValidationPal.GetRemoteCertificate and then CertificateValidationPal....

At this point the logic forks depending upon the OS.
CertificateValidationPal.VerifyCertificateProperties is in CertificateValidationPal.Unix.cs.

For Unix, this calls the CertificateValidation.BuildChainAndVerifyProperties.
This is in CertificateValidation.Unix.cs.

The X509Chain is built. It returns a ChainPal and runs BuildChain on it.
A linux ChainPal is returned.
It then runs the OpenSslX509ChainProcessor and runs BuildChain.
This has recursion as it enumerates the chain. It has a timeout to ensure the loading of certificates does not take too long.

The OpenSslX509ChainProcessor is run, and it runs the Pal.Unix/ChainVerifier.cs.
This runs HasUnsuppressedError which checks for errors that have not been surpressed. This includes checking for self-signed certificates.

The  Interop.Crypto.X509VerifyCert is called which calls calls native C code which calls the openSSL library function X509_verify_cert. This calls verify_chain.
OpenSSL build_chain will check for self signed certificates which has a return value of 18.

In OpenSSL, a self-signed certificate has a chain of 1.
Running 
openssl s_client -showcerts -verify 5 -connect cosmosdb.test:8081 
against a server will retun 

Debian ca-certificates notes.

Verify return code: 18 (self signed certificate)

In the dotnet core implhementation for Unix, Self-signed is when the subject name is the same as the issuer name (see code). Errors can occur because the peer trust is only supported for Windows certificate stores. Some of this is discussed here.

Having a self-signed certificate raises the X509VerificationFlags.IgnoreRootRevocationUnknown flag which needs to be surpressed, otherwise an error is returned.


This is a subtle difference from the standard Linux implementation of dotnet core; in dotnet core Self-signed is when the subject name is the same as the issuer name (see code).

If you have running Windows and a self-signed certificate in the Trusted Publishers store, then dotnet core will respect it and not throw an error. If you are running the same code on Linux and have installed the self-signed certificate into the ca-certificates store, it will throw an error.

The X509 Store differs by platform
The LocalMachine\Root store on Linux is an interpretation of the CA bundle in the default path for OpenSSL. The LocalMachine\Intermediate store on Linux is an interpretation of the CA bundle in the default path for OpenSSL. The CurrentUser\Intermediate store on Linux is used as a cache when downloading intermediate CAs by their Authority Information Access records on successful X509Chain builds.





Sunday, 10 February 2019

Docker commands & shortcuts

To stop all docker containers, in PowerShell:

docker ps -a -q | ForEach { docker stop $_ }

Using the Cosmos DB emulator across the network

It is very useful to be able to use the Cosmos DB emulator across the network; a good example being that you may be building Linux docker images to run on Kubernetes and you want them to access the Cosmos DB Emulator running on your development Windows desktop. Currently the Cosmos DB emulator only runs inside a Windows container, so running it as a container here won't help.

This article describes some of the steps to get you there.

I start off with installing the emulator on a development PC (the server) and getting a different Windows machine to access the emulator across the network. This proves the connectivity and then I move to configuring my Linux docker app.

CLIENT: Configure a hostname for accessing the emulator on the client PC
I am using the hostname cosmosdb.test to access the emulator.

In the client machine, running as an administrator edit the hosts file C:\windows\system32\drivers\etc\hosts and add the line:

<my emulator machine ip> cosmosdb.test

Note that the <my emulator machine ip> cannot be 127.0.0.1 as any running container will resolve this locally and try and connect to itself rather than the host machine. It has to be the host machines real IP address! (Another alternative would be to use the host.docker.internal hostname throughout).

Flush the local DNS to be sure (especially if you've run this step before)
ipconfig /flushdns

SERVER: Install and Run the emulator
Install the Cosmos DB Emulator.

Start it.

Stop it.

SERVER: Configure Windows Defender Firewall
Ensure Cosmos DB Emulator has a firewall exception for port 8081.

Start > Run > Windows Defender Firewall > Advanced Settings > Inbound Rules > New Rule

Select Port > TCP and Specific Local Ports > Enter 8081 > Allow the Connection

Select the networks to which the rule should apply. If you are really unsure, select Domain, Network & Public (but at your own risk!)

Give the rule and name and description and select OK.

SERVER: Generate a self-signed SSL certificate allowing the host name

Start the emulator from the command-line, adding the parameter
/GenCert=cosmosdb.test,<IP address>

Make a note of the certificate thumbprint (the last 5 digits may be sufficient).

SERVER: Export the SSL certificate
Start > Manage Computer Certificates

Right click on the root node and select Find Certificates

Change Look in Field to SHA1 hash and enter the last 5 digits of the thumbprint from the previous step into the Contains field.

You should see one (or more) entries for the Cosmos DB certificate.

Right-click on one and select Export.

Select No, do not export the private key.

Select Base-64 encoded X.509 (CER)

Save
to a file. We will save it as cosmosdbemulator.cer (and we'll refer to that filename later).

CLIENT: Import the SSL certificate on the client machine
On the client machine:

Start > Manage Computer Certificates

Select Trusted Root Certification Authorities > Certificates

Select All Tasks > Import > Next

Select the filename and click Next until the certificate is imported.

Double-click on the newly imported certificate, select the Details tab and confirm that the Subject Alternative Name property includes cosmosdb.test.

SERVER: Create a new access key

Start the emulator from the command-line, adding the parameter
/GenKeyFile=MyKeyFile.txt

Open the key file and make a note of the access key as this will be set explicitly when we start the emulator in future.

In my example I will use:
1rAUqgeKitqNhTKyKMtFQ7UK79d9cQhEfoiXomlh3zc1Qz58uQQV6c49B2wGeC9FuGM+OmRViCHRuJu3llb0Vg==

SERVER: Configure and start the emulator

The next steps configure the emulator for allowing access across the network. You may want to modify the emulator shortcut to set these command-line parameters, or run the executable directly from the command-line. Just remember, in future you must always start the emulator with certain additional parameters.

Start the emulator with the following parameters:
/NoFirewall /AllowNetworkAccess /Key=<access key>

or in my example
/NoFirewall /AllowNetworkAccess /Key=1rAUqgeKitqNhTKyKMtFQ7UK79d9cQhEfoiXomlh3zc1Qz58uQQV6c49B2wGeC9FuGM+OmRViCHRuJu3llb0Vg==


The AllowNetworkAccess must be set for the emulator to accept connections across the network.

CLIENT: Check connectivity
On the client PC open the command prompt and type
telnet cosmosdb.test 8081

And you should connect. If you don't then your host name is pointing to the wrong IP address or the firewall is still blocking you. If so, check the previous steps.

CLIENT: Run the test code
Run the test app on the client PC.

I have found that the V3 Beta Cosmos DB SDK works better when connecting across the network. I recommend you use this for all new development.

using System;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Azure.Cosmos;
using Newtonsoft.Json;

namespace cosmosdb_test_v3
{
    class Program
    {
        static void Main(string[] args)
        {
            var cts = new CancellationTokenSource();

            Console.CancelKeyPress += (s, e) =>
            {
                e.Cancel = true;
                cts.Cancel();
            };

            MainAsync(args, cts.Token).Wait(cts.Token);

            Console.WriteLine("Program exited");
        }

        private static async Task MainAsync(string[] args, CancellationToken token)
        {
            try
            {
                const string databaseId = "testDb";
                const string accountEndpoint = "https://cosmosdb.test:8081";
                const string accountKey = "1rAUqgeKitqNhTKyKMtFQ7UK79d9cQhEfoiXomlh3zc1Qz58uQQV6c49B2wGeC9FuGM+OmRViCHRuJu3llb0Vg==";
                const string containerId = "testCollection";
                const string partitionKeyPath = "/id";
                var documentId = "1";

                if (args.Length > 0)
                {
                    documentId = args[0];
                }

                var cosmosConfiguration = new CosmosConfiguration(accountEndpoint, accountKey);
                using (var client = new CosmosClient(cosmosConfiguration))
                {
                    CosmosDatabase database = await client.Databases.CreateDatabaseIfNotExistsAsync(databaseId, cancellationToken: token);
                    CosmosContainer container = await database.Containers.CreateContainerIfNotExistsAsync(containerId, partitionKeyPath, cancellationToken: token);

                    var person = new Person
                    {
                        Id = "Bloggs_" + documentId,
                        FirstName = "Joe",
                        LastName = "Bloggs_" + documentId,
                        Age = 35
                    };

                    await container.Items.CreateItemAsync(person.Id, person, cancellationToken: token);
                    Console.WriteLine("Document " + documentId + " created");
                }
            }
            catch (Exception e)
            {
                Console.WriteLine(e);
            }
        }
    }

    internal class Person
    {
        [JsonProperty("id")]
        public string Id { get; set; }
        public string FirstName { get; set; }
        public string LastName { get; set; }
        public int Age { get; set; }
    }
}

SERVER: Confirm that the document has been written
Open Data Explorer and confirm that the document has been written to the emulator.

Congratulations! You should now have run the dotnet core console application on one machine and connected to your development machine. Now we are going to run the same dotnet core console app as a Linux container.

Run Client Code in a Linux Docker container
Create a cert folder in the project folder and copy cosmosdbemulator.cer into that folder.
Rename the certificate to use a crt extension: cosmosdbemulator.crt.

Add the following Dockerfile to the root of the project:

FROM microsoft/dotnet:2.1-sdk
WORKDIR /app

# copy csproj and restore as distinct layers
COPY *.csproj ./
RUN dotnet restore

# copy the SSL certificate
ADD cert/* /tmp/cert/
RUN apt-get install ca-certificates && \
    cp -R /tmp/cert/cosmosdbemulator.crt /usr/local/share/ca-certificates && \
    update-ca-certificates

# copy and build everything else
COPY . ./
RUN dotnet publish -c Release -o out
ENTRYPOINT ["dotnet", "out/cosmosdb-test-v3.dll"]

Using the command-line in the root of the project, type

docker build -t cosmosdb-test-v3 .

Look for the text
Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.

to check that the SSL certificate has been installed correctly.

Note for update-ca-certificates to work the certificate should be found in the /usr/local/share/ca-certificates folder and have the .crt file extension. A different file extension (such as .cer) will not work.

Now run the test app container.

docker run --rm cosmosdb-test-v3 <my document number>

You can set any document number (just don't repeat a number or you will get a duplicate document exception!)

docker run --rm cosmosdb-test-v3 15

You should see
Document 15 created
Program exited

Congratulations! You now have a Linux docker container running the dotenet core Cosmos DB V3 SDK, writing a document to your host development emulator.

Debugging the certificates
If at any time you need to check the successful installation of the certificates on the Linux container you can run the container with a shell:
docker run --entrypoint /bin/bash -it cosmosdb-test-v3

You can check connectivity with:
curl -kv https://host.docker.internal:8081 and
curl -kv https://cosmosdb.test:8081

You can check the specific certificate would work if it were installed in the CA store using:
curl -v https://cosmosdb.test:8081 --cacert /tmp/cert/<mycert>.crt

Subsequent Notes
This works only if you connect your Cosmos DB client to another machine on the network; it doesn't work if you connect your Cosmos DB client to the same machine.

Friday, 11 January 2019

Turbocad activation

Turbocad activation can be done at the following URL:

http://activate.imsidesign.com

Thursday, 3 January 2019

Multiple partitions shown on FreeBSD when disk is formatted NTFS in Windows

A new 4TB USB disk was prepared for FreeBSD and formatted as NTFS using Windows 10.

The previous disk when connected was shown as /dev/da1s1
However the new device had to be mounted as /dev/da1p2

Trying to mount with
ntfs-3g /dev/da1p2s1 /mnt/usb
threw the error
The device '/dev/da1p2s1' doesn't seem to have a valid NTFS.

Why the difference?

Looking at gpart show /dev/da1 it indicates

=>        34  7814037100  da1  GPT  (3.6T)
          34      262144    1  !e3c9e316-0b5c-4db8-817d-f92df00215ae  (128M)
      262178        2014       - free -  (1.0M)
      264192  7813771264    2  !ebd0a0a2-b9e5-4433-87c0-68b6b72699c7  (3.6T)
  7814035456        1678       - free -  (839K)

The first partition is a 128M Microsoft Reserved Partition.
The usable data exists in partition 2.

To mount the drive for copying data use:
mkdir /mnt/usb
kldload fuse
ntfs-3g /dev/da1p2 /mnt/usb
rsync -av --ignore-errors --exclude .recycle /mnt/Data /mnt/usb