I woke up to a message on my phone, "The system is not working". Gulp, we only have one system so if it isn't working, that's pretty serious.

Ignoring the red-herring caused by Virgin media returning random IP addresses to DNS requests, I realised that the client certificate used for authentication between web app and web service had expired. I didn't actually realise this was an issue for client auth, although it is fair enough. I then scrambled around for an hour or so trying to fix it, and test it and deploy all the changes. Here is what I learned that should save you some time!

Note: You will often need to use mmc.exe to manage certificates. When you run it, choose File->Add/Remove snap-in->certificates and either choose Local Machine or Current User (or both if you need to).

  1. You need a cert that has not expired!
  2. Your certificate needs to have Client Authentication as one of its permitted uses.
  3. It needs to be in the relevant store (usually LocalMachine/My, which needs to be matched on your local machine to test it with).
  4. You might need to give permission for all users to read the private key of the cert in mmc.exe depending on what testing you are doing.
  5. The certificate needs to chain to a root certificate. Theoretically, you could do this on Azure by installing your own root cert on the cloud service but this is not directly supported by Visual Studio and would need to be done via Powershell or similar in the startup script.
  6. You need to make sure that you have uploaded the new certificate into the certificates tab of the portal (for app and web service).
  7. You need to reference the new certificate in the Azure project (role) settings so that Azure installs it into the cloud instance from the certificates tab.
  8. You need to change the thumbprint in the service settings of the web service to reference the new thumbprint. It is easiest to copy this from the Azure portal because the thumbprint is displayed in a single block of text.
  9. Upload the modified web service and access it from the browser. You should be offered a dialog to select a client cert, choosing it should allow you to access the svc of the web service. If it is not in the list you are shown, it is either expired, not present in the store for the Local User or does not have Client Authentication as one of its uses. It might be caused if the certificate does not chain on your local machine, you can check this in mmc.exe by double-clicking the certificate and choosing the Certification Path tab.
  10. You will probably need to refresh the service references for the web app (see below).
  11. If you have used Windows credential manager for client certs, you will need to update this to use the new cert otherwise svcutil.exe will fail with 403 (forbidden).
  12. Refresh the service references in the web app.
  13. Change the thumbprint for the client cert in the web app (probably web.config).
  14. If you can test the web app locally, it will save you the upload time to find out if it doesn't!
  15. If you get a 403 at any point, it means the certificate cannot be founded or read. This might mean the permissions are not correct, the certificate is not in the correct store (remember to differentiate between CurrentUser and LocalMachine) or you do not have permissions to read it from the store.