WCF date time serialization

I just encountered a horrible problem where my WCF service was doing all the hard work calculating hourly volumes for long and short days with respect to Daylight Savings Time. Results were being returned as two properties, StartTimeLocal and StartTimeUtc.

My unit tests all worked fine.

But when the clients were consuming the service a problem was being encountered. The times were different and causing problems.

In the end I realised - because the DateTime was defaulting to DateTimeKind.Local, the client was taking the property named UTC, assuming it was local time and adjusting it!

The rule is - it is safest to return your DateTimes as DateTimeKind.Utc - that way the client will not make assumptions about timezone conversions.

SSIS Merge Not Working - sort order

I discovered an error scenario with SSIS merge joins.

I had a merge join on two datasets, based upon a join on two columns: COL_A and COL_B.
The merge join tool requires the inputs to be sorted, which they were.

However the change I made required a change to the order sequence.
Instead of it being Col_A = 1, Col_B = 2 it was changed to Col_B = 1, Col_A = 2.

The order sequence was specified logically in SSIS properties of the input data sources. However the SQL query for the 2nd data source was not updated, so it was left as

ORDER BY COL_A, COL_B

No errors were reported as they were both of type INT. However the JOIN failed and rows were being excluded that were present before.

So it is important to ensure that the data is PHYSICALLY sorted (either by using the SORT tool) or by the SQL query itself, not just sequence it in the properties of the data source. This is noted here with a warning:

If the sort options indicate that the data is sorted, but the data is not actually sorted, the results of the merge or merge join operation are unpredictable.

 It's just a bit nasty because if you make this mistake, there are no warnings or errors and the first indication that something is wrong is when the output isn't what you expect.

You can also conclude that an SSIS join is different from a SQL join - it is dependent upon column positions rather than column names.

Self-hosted WCF service, using Windows Authentication, running under a user account

So here is the summary.

If you have a self-hosted WCF service, running under a user account, authenticating clients using Windows Authentication, which is coded manually (not using configuration) you must follow the rules in order to avoid the dreaded 401 Unauthorized errors.

1. The service endpoint must specify a UPN

  var endpointAddress = new EndpointAddress(
                new Uri(ConfigurationManager.AppSettings["WebServiceUrl"]),
                EndpointIdentity.CreateUpnIdentity(WindowsIdentity.GetCurrent().Name));

2. The client needs to connect using the same UPN. In other words it needs to pass the user account name that the server is running under.

var channelFactory = new ChannelFactory<IVolumeService>(
                binding2,
                new EndpointAddress(new Uri(connection.Url) 

, EndpointIdentity.CreateUpnIdentity("the user account name that the server is running under")));

If you generated a normal service with a Metadata endpoint, you would see this in the WSDL:

<Identity xmlns="http://schemas.xmlsoap.org/ws/2006/02/addressingidentity">

  <Upn>APOTTS@MyDomain.intra</Upn>

</Identity>

And if you generated a client from the WSDL you'd see this hidden somewhere in the client proxy.

The UPN would be the user name of the domain account under which the server was running. The server validates what the client sends to ensure they can both trust each other before it tries to raise a Kerberos ticket.

WCF - System.Net.WebException: The remote server returned an error: (401) Unauthorized

Further to my previous post about SPNs and WCF applications, there is a further twist in the tale.

As I reminder, I discussed beforehand how I had a self-hosted WCF application that use Windows Security as the Authorisation mechanism. When I deployed this to a remote server, clients could not authenticate correctly unless they had the SPN set in the client code:

var channelFactory = new ChannelFactory<IVolumeService>(
                binding2,
                new EndpointAddress(new Uri(url), new SpnEndpointIdentity("MYSERVICE/MyMachine")
                    ));

Without this setting they would return a 401 Unauthorised, with the detail:

The HTTP request is unauthorized with client authentication scheme 'Negotiate'.

The authentication header received from the server was 'Negotiate oW8wbaADCgEBom

YEZGBiBgkqhkiG9xIBAgIDAH5TMFGgAwIBBaEDAgEepBEYDzIwMTIwNjI5MTY1NjQ1WqUFAgMOYYqmAw

IBKakRGw9HQVpQUk9NVUsuSU5UUkGqEzARoAMCAQGhCjAIGwZhcG90dHM='.

The remote server returned an error: (401) Unauthorized.

The target principal name is incorrect

Adding the SpnEndpointIdentity allowed the clients to work with this remote server.

However, the twist was this: when I went back to working locally, with a local client and a local server running under my current user (with admin privileges) account - it no longer worked!

This time I would get

System.Net.WebException: The remote server returned an error: (401) Unauthorized

.

   at System.Net.HttpWebRequest.GetResponse()

   at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpCha

nnelRequest.WaitForReply(TimeSpan timeout)

Note there was no The target principal name is incorrect message. It took me a while to figure it, but in the end REMOVING

new SpnEndpointIdentity("MYSERVICE/MyMachine")

got it to work locally! Bah! More reading on SPNs is required.

....

Double Bah! I figured it. It was because I had left a bit of code on the server

var endpointAddress = new EndpointAddress(
                new Uri(ConfigurationManager.AppSettings["WebServiceUrl"]),
                EndpointIdentity.CreateSpnIdentity("HOST/servername.domainname:9011"));

and clearly the SpnIdentity in this mismatched the client. Setting BOTH to be the same caused it to work again.

No, this still didn't work. The solution was to use a UPN rather than a SPN, because the account was running under a user account and not localnetwork or system. Both server and client used:

 var endpointAddress = new EndpointAddress(
                new Uri(ConfigurationManager.AppSettings["WebServiceUrl"]),
                EndpointIdentity.CreateUpnIdentity(WindowsIdentity.GetCurrent().Name));

A hint was also shown by looking at the WSDL of an existing endpoint that ran under a user account:

  <Identity xmlns="http://schemas.xmlsoap.org/ws/2006/02/addressingidentity">
  <Upn>MYUSER@mydomain.intra</Upn>
  </Identity>

Lessons learnt:

1. Self hosting WCF services is more painful than hosting them in IIS.
2. Try and use the configuration approach as much as possible - there are more examples out there.
3. If you don't publish a metadata exchange you have to bind manually, and this provides more chances to mismatch.
4. Programmatically defining and consuming the endpoints is less well documented - and self hosting with Windows Authentication and SPNs is sparsely documented!
5. Generating your client from the metadata exposed from the server is much safer. 

References:
http://msdn.microsoft.com/en-us/library/ms733130.aspx
http://msdn.microsoft.com/en-us/library/ms677601%28v=vs.85%29.aspx
http://social.msdn.microsoft.com/forums/en-US/wcf/thread/78638457-ca7a-4f88-b8a9-9bc32d4b5c7d/
http://msdn.microsoft.com/en-us/library/bb628618.aspx

Certificate Bugs - Certificate add failed, Error: 1312 A specified logon session does not exist. It may already have been terminated.

Welcome to my world - simple things that should work fail miserably due to bugs in products.

Problem
This time I was attempting to add a certificate to a port in order to allow a self-hosted WCF service to work over HTTPS. Most examples are about IIS, which is simpler.

The command was this:

netsh http add sslcert ipport=0.0.0.0:9011 certhash=80f06fa16c2dee8abccddbcb1c0694e7e0c5ffdd appid={08AAC041-869A-4C12-AF5B-999D0F4ABC43}

but it was returning an error:

Certificate add failed, Error: 1312 A specified logon session does not exist. It may already have been terminated.

There are lots of blog entries out there, but in the end I found my problem was due to import bugs in MMC.

There are a couple of checks to make first though:

1. The certificate that you are using to run the HTTPS MUST have a private key.
Therefore when you are creating the certs, create them with the following commands:

makecert -sk testRootCA -sky signature -sr localmachine -n "CN=RootTrustedCA" -ss TRUST -r RootTrustedCA.cer (to make the root cert)

makecert -sk testServer -ss MY -sky exchange -sr localmachine -n "CN=Server" -ic RootTrustedCA.cer -is TRUST Server.cer -pe (to make the HTTPS server cert signed by the previous)

2. Check whether the hotfix is applicable (it wasn't for me).

3. Try this excellent tool and see whether it gets round the 1312 error (though it is unlikely to if you had the same problem as me).

Solution
Create the certificates as described above. Ensure the server certificate has the private key.

Then I imported the RootTrustedCA into the Trusted Root Certification Authorities store. NOTE - you must do this for the Local Computer account when you open MMC.

Then I imported the Server certificate into the Personal Certificates store.

But here is the thing. If you look at the certificate at this point, it is missing the key symbol indicating that it has a private key. Somehow it has lost the private key.
How did I fix this? Simple, run the makecert command again, and refresh MMC and you'll see that the certificate has gained it's private key. There is obviously some horrible bug there.

Now double-click on the certificate and grab the thumprint. Enter this into the netsh command.

netsh http add sslcert ipport=0.0.0.0:9011 certhash=80f06fa16c2dee8abccddbcb1c0694e7e0c5ffdd appid={08AAC041-869A-4C12-AF5B-999D0F4ABC43}

and hopefully you'll get:

SSL Certificate successfully added

Useful tools

ILSpy - dissasmble .NET assemblies (think Refactor, but open source)
ConfuseEx - open source .NET obfuscator
Dotnet IL Editor
SOAPUI
Fidder - acts as a system wide proxy
Squid - proxy server
WinDirStat - disk usage viewer
SlickRun - a tool for quickly starting programs
ZoomIt - useful for presentations and screen sharing
Inkscape - SVG editor
Charles Debugging Proxy
GIMP - Good free image program
Handbrake - Video transcoding (for ripping DVDs)
DVD Decrypter - for ripping DVDs
OBS Studio - Open Source streaming and broadcasting
User Agent Switcher (for Firefox)

Macrium Reflect - Disk Cloning software

PDF Architect used to be good, now they've ruined it. An acceptable alternative is PDF Split and Merge.

https://etcher.io burn sd cards
EaseUS partition master - the free version allows you to manage HDDs up to 8TB.

ImageMagick - Batch conversion of images (e.g. Apple HEIC to JPG)

Open Source video editors

The Register

Openshot

Shotcut

Pitivi

KdenLive

Cinlerra

Flowblade (Linux)

Lumiera

Blender (3D creation)

Avidemux (simple)

Lightworks (free is 720p only)

Video Recording  and Streaming

OBS Studio

Developer Tools

Dependencies - an open source alternative to dependency walker (depends.exe)

Network Tools

6 Free Tools to Test and Benchmark Your Network Speed • Raymond.CC

Radio tools

https://github.com/merbanan/rtl_433/ - decoding 433MHz

https://github.com/jopohl/urh - universal radio hacker

WCF and Windows Authentication - The HTTP request is unauthorized with client authentication scheme 'Negotiate'

I have a simple WCF service hosted with in Console application, secured by Windows Authentication.

    <services>
      <service name="MyService.Service1" behaviorConfiguration="MyServiceTypeBehaviors" >
        <endpoint contract="IMetadataExchange" binding="mexHttpBinding" address="mex" />
        <endpoint address="http://localhost:9001/MyService"
                  binding="basicHttpBinding"
                bindingConfiguration="BasicHttpEndpointBinding"
                  contract="MyService.IService1" />
      </service>
    </services>
    <bindings>
      <basicHttpBinding>
        <binding name="BasicHttpEndpointBinding">
          <security mode="TransportCredentialOnly">
            <transport clientCredentialType="Windows" />
          </security>
        </binding>
      </basicHttpBinding>
    </bindings>

When the client calls the service, it fails and is returned the following exception message:

The HTTP request is unauthorized with client authentication scheme 'Negotiate'.
The authentication header received from the server was 'Negotiate oW8wbaADCgEBom
YEZGBiBgkqhkiG9xIBAgIDAH5TMFGgAwIBBaEDAgEepBEYDzIwMTIwNjI5MTY1NjQ1WqUFAgMOYYqmAw
IBKakRGw9HQVpQUk9NVUsuSU5UUkGqEzARoAMCAQGhCjAIGwZhcG90dHM='.
The remote server returned an error: (401) Unauthorized.
The target principal name is incorrect

 The client config was as follows:

<system.serviceModel>
    <bindings>
      <basicHttpBinding>
        <binding name="BasicHttpBinding_IService1" closeTimeout="00:01:00"
            openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"
            allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard"
            maxBufferSize="65536" maxBufferPoolSize="524288" maxReceivedMessageSize="65536"
            messageEncoding="Text" textEncoding="utf-8" transferMode="Buffered"
            useDefaultWebProxy="true">
          <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
              maxBytesPerRead="4096" maxNameTableCharCount="16384" />
          <security mode="TransportCredentialOnly">
            <transport clientCredentialType="Windows" proxyCredentialType="None"
                realm="" />
            <message clientCredentialType="UserName" algorithmSuite="Default" />
          </security>
        </binding>
      </basicHttpBinding>
    </bindings>
    <client>
      <endpoint address="http://edevd3ap2lonuk:9001/MyService" binding="basicHttpBinding"
          bindingConfiguration="BasicHttpBinding_IService1" contract="ServiceReference1.IService1"
          name="BasicHttpBinding_IService1">
        
      </endpoint>
    </client>
  </system.serviceModel>

This config seems normal and is similar to all the examples online. It should be noted however that most of the online examples are weighted towards a service hosted in IIS whereas this (due to technical restrictions) had to be hosted within a Console application or Windows Service.

The first lesson with WCF errors is to look carefully at the inner exceptions, and not just the exception message but at the headers part too. The clue in this error was the message "The target principal name is incorrect". The first hint came from here.

Additional reading included Security in Windows Communication Foundation and Service Identity and Authentication.

Overriding the service identity is described here:
http://msdn.microsoft.com/en-us/library/bb628618.aspx

Note that it also describes the behaviour when the Service Principal Name is missing:

When SPN or UPN Equals the Empty String

If you set the SPN or UPN equal to an empty string, a number of different things happen, depending on the security level and authentication mode being used:

• If you are using transport level security, NT LanMan (NTLM) authentication is chosen.
• If you are using message level security, authentication may fail, depending on the authentication mode:
• If you are using spnego mode and the AllowNtlm attribute is set to false, authentication fail.
• If you are using spnego mode and the AllowNtlm attribute is set to true, authentication fails if the UPN is empty, but succeeds if the SPN is empty.
• If you are using Kerberos direct (also known as "one-shot"), authentication fails.

Solution

The issue was to do with the fact the client did not have a Service Principle Name added.

Modifying the client config to include this fixed the issue:

<system.serviceModel>
    <bindings>
      <basicHttpBinding>
        <binding name="BasicHttpBinding_IService1" closeTimeout="00:01:00"
            openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"
            allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard"
            maxBufferSize="65536" maxBufferPoolSize="524288" maxReceivedMessageSize="65536"
            messageEncoding="Text" textEncoding="utf-8" transferMode="Buffered"
            useDefaultWebProxy="true">
          <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
              maxBytesPerRead="4096" maxNameTableCharCount="16384" />
          <security mode="TransportCredentialOnly">
            <transport clientCredentialType="Windows" proxyCredentialType="None"
                realm="" />
            <message clientCredentialType="UserName" algorithmSuite="Default" />
          </security>
        </binding>
      </basicHttpBinding>
    </bindings>
    <client>
      <endpoint address="http://edevd3ap2lonuk:9001/MyService" binding="basicHttpBinding"
          bindingConfiguration="BasicHttpBinding_IService1" contract="ServiceReference1.IService1"
          name="BasicHttpBinding_IService1">
        <identity>
          <servicePrincipalName value="MYSERVICE/MyMachine"/> 
        </identity>
      </endpoint>
    </client>
  </system.serviceModel>

Almost always the examples are in code which is a little difficult if you need to do it in code. The code solution, using a ChannelFactory, is below:

var client =

channelFactory.CreateChannel(new EndpointAddress(new Uri("http://edevd3ap2lonuk:9001/MyService"), new SpnEndpointIdentity("MYSERVICE/MyMachine")));