Sunday 30 October 2016

Updating the Galaxy Tab S to support adopted storage

Grrr. Samsung have purposfully disabled adopted storage for the Galaxy Tab S, meaning that all of the 16GB rapidly fills, especially when app developers lazily prevent the app to be moved to external storage.

I tried connecting the tablet via USB and running the
adb shell sm set-force-adoptable true
command, but this doesn't work.

There are steps to configure Adopted Storage which involves TWRP and the Storage Enabler. And it appeared to work in large - after following the steps I had the option to install a card as internal memory. But I had a random error on the Settings screen after formatting the SD card and also when copying files to the phone it still said "Out of disk space" error. At that point, after 3 hours of trying I gave up and got on with my life.

Lessons learned:
Samsung create great phones and tablets, but the firmware sucks. And there isn't a phone out there at the moment that supports adoptable storage and a replaceable battery, and no tablet supporting adoptable storage. It's as if they want you to have their products with a very short usefulness lifespan, if not a physical lifespan.

References
https://nelenkov.blogspot.co.uk/2015/06/decrypting-android-m-adopted-storage.html?view=flipcard
http://forum.xda-developers.com/galaxy-tab-s/general/patch-adoptable-storage-enabler-t3460478/page4
http://odindownload.com/
http://www.modaco.com/news/android/heres-how-to-configure-adoptable-storage-on-your-s7-s7-edge-r1632/

Updating the Galaxy Tab S to support adopted storage

Grrr. Samsung have purposfully disabled adopted storage for the Galaxy Tab S, meaning that all of the 16GB rapidly fills, especially when app developers lazily prevent the app to be moved to external storage.

I tried connecting the tablet via USB and running the
adb shell sm set-force-adoptable true
command, but this doesn't work.

There are steps to configure Adopted Storage which involves TWRP and the Storage Enabler. And it appeared to work in large - after following the steps I had the option to install a card as internal memory. But I had a random error on the Settings screen after formatting the SD card and also when copying files to the phone it still said "Out of disk space" error. At that point, after 3 hours of trying I gave up and got on with my life.

Lessons learned:
Samsung create great phones and tablets, but the firmware sucks. And there isn't a phone out there at the moment that supports adoptable storage and a replaceable battery, and no tablet supporting adoptable storage. It's as if they want you to have their products with a very short usefulness lifespan, if not a physical lifespan.

References
https://nelenkov.blogspot.co.uk/2015/06/decrypting-android-m-adopted-storage.html?view=flipcard
http://forum.xda-developers.com/galaxy-tab-s/general/patch-adoptable-storage-enabler-t3460478/page4
http://odindownload.com/
http://www.modaco.com/news/android/heres-how-to-configure-adoptable-storage-on-your-s7-s7-edge-r1632/

Galaxy Tab S stuck in "Downloading" mode

Galaxy Tab S stuck in "Downloading" mode:

Press: Power + Vol Down + Home for 20 seconds

Recovery Mode
Turn off the device
Press and hold Volume UP key + Home Key
then Press and hold Power key
Release all key when you see Android System Recovery
Use Volume UP key and Volume Down key to select Menu
Use Power key to Confirm or Execute Menu

Download Mode or ODIN Mode
Turn off the device
Press and hold Volume Down key + Home key
then Press and hold Power key
Release all key when you see ODIN Mode
Use Volume Up key to continue
Use Volume Down key to cancel ( Restart the device )

Odin is a tool for updating the firmware on Samsung devices.

Monday 17 October 2016

Distributed Systems: de-duplication and idempotency

When working on distributed systems there are two important concepts to keep in mind - deduplication and idempotency. The two are sometimes confused but there are subtle differences between them.

Background Context
In our e-commerce system, we receive orders submitted from Checkout and manage them through a lifecycle of Payment Authorisation, Fraud Check, Billing through to Dispatch. We employ asynchronous message-driven architecture using a combination of technologies such as NServiceBus, MSMQ and Azure Service Bus. We use a combination of workflow choreography and orchestration. In this lightweight handlers receive a command or subscribe to an event, do some work, and publish a resulting message, These are chained together with other handlers to complete the overall workflow.

At-Least Once Messaging
Messaging systems typically offer at-least-once messaging, That means you are guaranteed to get a message once but under certain circumstances - such as failure modes - you may get the message twice (perhaps on different independent threads of execution).

Avoiding certain work twice
However, some work items should not be repeated: you should only charge a customer once or perform a fraud check once. If our handler receives a PaymentAccepted event it performs a Fraud Check with a third party service. This costs money and repeatedly calling it will affect a customers fraud score - it should only be done once per order.

Deduplication
Some people employ deduplication to solve this. For example, Azure Service Bus gives you the option to deduplicate messages by setting a time window (say 10 minutes). In this Azure Service Bus will check every message ID and if it sees it has already been processed it will prevent another handler processing it. This is possible because ultimately the broker architecture revolves around a single SQL database (limited to a region).

However, deduplication does not solve all of the problems. Idempotency is very important too,

Idempotency
If we are truly idempotent then a handler should “always produce the same output given the same input”.

Ideally, a handler would be fully idempotent. If the work was to set subscribe to a NewCustomerOrder event and set a new customer flag to true and publish a NewCustomerUpdated event, we should be able to publish this event ten times and the outcome will always be the flag being set to true and a NewCustomerUpdated flag published.

However, in the Fraud Check scenario, we should subscribe to PaymentAccepted event and check history to determine whether to perform a Fraud Check, and always publish the result - normally FraudCheckPassed.

Idempotency supports Replays
This is important for replays. We had a system problem where our system experienced a series of failures that caused messages to be lost at various stages of the workflow. 

This is shown below:


It would have been very useful to go to the leftmost part of the workflow and replay the message: this would have resulted in the downstream handlers responding and repeating their work as required. However many handlers employed deduplication: they simply swallowed the input event, did no work, and published no outcome. That meant latter stages of the workflow were not exercised and the system had to be recovered stage by stage. It required a lot of manual effort, was time consuming and was prone to error.

If each handler was idempotent; it would have received the input event, chosen whether to do work or not, and published the output event.

What are the side effects?
So impact could this have?

In the event of duplicated messages at the transport level (e.g. at least-once messaging) we could end up with more load on the system. If one of the leftmost handlers received a duplicate, it would propagate down the right. However I believe this is a small price worth paying for the enhanced supportability. Furthermore Azure Service Bus, being a broker with message-level locking, will prevent this except for handler failure scenarios.

If we are using Event Sourcing then if we duplicate or replay an event our event streams will record the fact that the FraudCheckPassed was published twice. Arguably, though, this is a good thing. It is a true reflection of history, which is what the event stream should be. It is much better to have two publishes recorded rather than someone manually hacking the event stream to fix problems.

In our current system, we enforce deduplication at the transport layer using Azure Service Bus configuration. However I believe this should be changed anyway; it prevents partitioned queues, it prevents us changing transport layer and anyway the deduplication and idempotency should be enforced at the handler level. 

Deduplication
Note that idempotency and deduplication are separate concerns (although often mixed!). In the event of a Fraud Check or a call to a Payment Processor for billing it is important that these calls are not repeated twice. If we receive a duplicate or replayed message, we don’t call these external systems again. Rather we just republish the last outcome.

Closing argument
Our APIs are idempotent so why aren’t our handlers? When an API receives the same input request, it publishes the same output. This is important if the client enters a tunnel when issuing an HTTP request.
Why aren’t are handlers behaving in this manner?

Tuesday 11 October 2016

Netscaler Gateway Plugin

The Netscaler Gateway plugin is incompatible with VirtualBox's "VirtualBox Host-Only Ethernet Adapter". You get timeouts when you do a ping test.
When the latter is enabled, the VPN connection intermittently fails to route traffic correctly. Disable the adapter to work with the Netscaler gateway.

Saturday 8 October 2016

gpedit is the new control panel: Windows Update and Windows Ink.

Today I got rather annoyed by Windows 10. I know it is supposed to cater for a wide range of users including those whom are non-IT savvy, but it has taken "knowing best" to an all new level.

When I turned on my laptop today:

- it spent ages installing a new update
- it overwrote my Windows + W shortcut to launch a new program, Windows Ink Workspace, which I don't want
- it then tried to reboot as I was working

I then spent half an hour trying to find the option to uninstall Windows Ink. You can't. I then tried to find it's option to release my Windows + W shortcut. You can't.

This is too aggressive. I like my computer they way I had it before the update. I don't expect someone to come into my office, replace my pens and chair, and rearrange my desk because they are think they are doing me a favour. I want it like I had it.

So can you reconfigure your Windows 10 to turn off all this rubbish? To ask before installing updates and disable Windows Ink Workspace? Well, no. Not using the front door.

Fortunately there is gpedit. It appears this is becoming the new control panel. The old control panel is for the non-IT savvy users who just want to do the basics.

So to stop Windows Ink Workspace:
http://www.thewindowsclub.com/disable-windows-ink-workspace

To force Windows 10 to be polite and ask before downloading updates over your Internet connection:
http://www.howtogeek.com/224471/how-to-prevent-windows-10-from-automatically-downloading-updates/