This is a tale of woe and wasted time that relates to a problem that Microsoft have had for a long time and have not really improved in Visual Studio. It mostly relates to library references and to be fair, some of the problems are related to third-party vendors who make things fail silently but Visual Studio has become so static that simple improvements are not made and us .net Devs are watching VS hang, take a long time to do things, make unwelcome connections to the internet synchronously and a lot of other horrors.

I think that if Microsoft persist in making big strides without looking at the details and annoyances, then they will start to lose market, particularly to JS frameworks which are starting to look decent enough to replace .Net code in many instances as well as Node JS which works pretty well for back-ends. Both of which can be run in very fast editors that don't do weird or clever stuff.

Back to the story. OWIN startup wasn't being called (TL;DR: I was missing a nuget package for something related to code called in Startup, NOT owin stuff itself). This is a project that has been running fine but recently, we have had a number of issues related to versions in build and those icky messages about resolving versions and everything, so we decided to remove all references, change the project to package reference and add back in those that we needed. This should have been clean and easy but there is an issue when .Net framework reference net standard libraries: they can pull in the direct dependencies but not the dependencies of dependencies so you need to be really careful that you have everything you need. The best way to do this is compare the Bin directory before and after and make sure they match!

Anyway, I noticed that Startup.Configuration wasn't called and had to go back and see what I had removed in packages.config. I added things back in and over many hours of trial and error, I added a missing EntityFramework nuget package and it started being called! (Say what?). That's right, a missing dll did not cause a compiler or a runtime error, it caused Startup.Configuration to not be called! This was completely counter-intuitive but at least I had managed to work it out...

...until it got to the build server and was failing again!

Very long story short, we were missing a nuget package which was related to something called in Startup and it had caused Startup to NOT be called at all. No error, no nothing just a ton of trial and error and going backwards to add in a tonne of stuff I might not need just to find out what happened.

At least I had managed to work it out...

...until it got to the dev web server. Same problem.

I am out of options now. The dlls match between build and dev but it doesn't work. I have raised an issue on github but this brings me to my gripes about Visual Studio builds:

  1. Clean doesn't clean! We know this. It deletes what it is expecting to write and leaves everything else. This is bad and should have no place in a build tool. If you have dlls that cannot be cleaned, use a build step to copy them in after it has cleaned.
  2. Rebuild doesn't always rebuild.
  3. The Fusion loader helpfully looks in loads of places for references, whether you want it to or not. "Great, found it in the GAC" but wait until you get deployed. It should not do this except for system dlls and ONLY if there are guaranteed deployed with the net framework runtime i.e. guaranteed to be present on a deployed machine.
  4. You are allowed to directly references libraries in the packages folder without using nuget package references. I'm not sure whether part of this is VS crappy "Hintpath" parameter but this is just another way for builds to not work correctly if you check out into a different structure. It also places depedencies onto the projects that are populating the packages folder so clean and rebuild and it doesn't work.
The net result is that it is very hard to make consistent builds between dev machines and build servers. Visual Studio should be much harder on correctness than this:

  1. Big loud warning if you browser to a library in the packages folder
  2. Do not find reference in the GAC unless you tell it to
  3. Clean should clean everything
  4. Hint path is rubbish, there should be a proper way to locate references 
  5. Errors need to be logged by everything that expects something that doesn't work. Even if third-party tool expects something that might not be present for a good reason, it can still log that message
  6. ASP Startup needs to allow quality verbose logging of what's happening but not the crazy level with 10GB of machine dumps but instead more like the fuslogvw produces: what was expected, what was searched, what happened etc.
Then I wouldn't have wasted over a day on an app that still doesn't work!