Ah, Fridays. The time of relaxation, chilling and really annoying bugs that should be easy to understand and track down but not! In this case, I pretty much knew the cause but I didn't know how to diagnose exactly what was causing the problem.

The Bug

I ran my unit tests locally and they ran fine. I uploaded to the build server and got the error above. The same problem could happen for any number of libraries and methods, the error being something like this: System.MissingMethodException: Method not found: 'Void Microsoft.Azure.KeyVault.KeyVaultClient..ctor(AuthenticationCallback, System.Net.Http.DelegatingHandler[])'.

The Basic Problem

If it builds OK, then the references are working correctly. Your compiler has found the method, but, of course, only links symbolically to an assembly with a version (in this case System.Net.Http.dll 4.2.0.0). At runtime, however, it uses the DotNet assembly loader, which you have probably learned is a pain in the neck because there can be multiple versions of assemblies on one machine and not another as well as assembly redirects, dependencies of dependencies and different framework versions.

It loads whatever library the other loader tells it to and if that doesn't have the method (or if the types in it are found in a different version of a dll it assumes it is different) you get the error.

In my case, the only real clue was that it worked locally and not on the Build Server. Fortunately I had access to both.

Diagnosing

The first thing that is important is that you shouldn't believe what Visual Studio tells you about versions. When I right-clicked the method that couldn't be found in my code and went to the definition and then to another dll that had another dependency in, it told me it was in System.Net.Http.dll 4.2.0.0 from the SDK install, which made sense and which should be correct.

This was not the case at runtime. I will spare you the things I tried but the first thing is to run fuslogvw on the local machine and log all bindings to disk. This tool is critical for debugging things like this! You should run it as Administrator from a command prompt and then in settings, enable all bindings and set a custom path (e.g. c:\temp\fusion). Then run the task that works locally but not on the build server and then go back to fusion log and disable the logging, just to reduce noise!

You should then look through and find the assemblies of interest, which are listed by the version being requested by the app, in my case 4.2.0.0:


Double-click this entry and you will see an html file with the loading details in it. Note that the order is the order they are processed so you might see the same item twice in the list. In my case, the issue was in the unit test project so I started at the bottom of the list and scrolled up until I found the requested version.

On my local machine, this file contained something very interesting (snipped):

LOG: Version redirect found in framework config: 4.2.0.0 redirected to 4.0.0.0.
LOG: Post-policy reference: System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
LOG: Switch from LoadFrom context to default context.
LOG: Binding succeeds. Returns assembly from C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Net.Http\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Net.Http.dll.
LOG: Assembly is loaded in default load context.

Note the first line that says although I requested 4.2.0.0, a redirect was changing this to 4.0.0.0, which was in the Gac and which was loaded! Not the 4.2.0.0 that Visual Studio thought it was using.

Looking at the Build Machine

Well, I did exactly the same thing on the build machine with fuslogvw and sure enough, it was loading the expected 4.2.0.0 and not 4.0.0.0. Ironically, the Build Server seemed more correct but was not working!

There were two obvious questions. 1) Why was my local machine redirecting the assembly and 2) Why did it even matter?

I tried a find-in-files on my local machine and could not find an answer. I could not find the assembly redirect so I decided to look into the second question, which ended up partially answering the first.

The Microsoft Cock-up

Microsoft, I love you but sometimes you make basic errors that affect thousands of people, especially with versions and packaging. One such example, I found on a massive thread here.

Basically, when dotnet core was released, there was more work needed to the main DotNet assemblies to move shared types into the correct place and move out platform-specific stuff into platform-specific libraries. Changes which were not really needed for normal .Net Framework and so the decision was taken to simply continue the development of System.Net.Http.dll for .net core without updating the .Net framework version (which was still 4.0.0.0). These versions had changes, including breaking ones, and also updated the version numbers for the .net framework library even though it was not changed (to keep it in line with the .net standard version) causing all manner of dependency problems, especially where .net standard packages were referencing the NuGet package for this library instead of just referencing it directly (why is the package even there?).

The suggested fix for most problems was simple: redirect anything >4 back to 4.0.0.0, which is the version in the Gac. This explains why it was happening on my local machine but not how. It also explained why the build was failing on the Build Server, which didn't have the redirect in place.

The Solution

Once all of that was understood, the solution was simple: An assembly redirect in the app.config for the test project and it was all happy!

<dependentassembly>
    <assemblyidentity culture="neutral" name="System.Net.Http" publickeytoken="b03f5f7f11d50a3a">
        <bindingredirect newversion="4.0.0.0" oldversion="0.0.0.0-4.2.0.0"></bindingredirect>
    </assemblyidentity>
</dependentassembly>