There is a software principle that you should not attempt to deploy more than one new thing at the same time because if it doesn't work, you won't know where to start! This is a system that is being primed for production but isn't production yet so that's OK.

What was I doing? Adding a healthcheck endpoint to a service running in a dotnet core container running on Linux on Azure Kubernetes Service behind an nginx ingress. The healthcheck is calling into a database whose connection string comes from a Kubernetes secret!

This is what you need to do:
  1. You need to make sure that your nginx ingress (as described here) is setup to forward the client ip by adding another switch to Helm. This means you have to terminate SSL at the ingress but you should be doing that anyway! If you already have the ingress, you can call Helm upgrade but BE WARNED if you call set without adding --reuse-values to the command, it will remove the existing custom parameters and add in the new ones!
  2. Dotnet core behaves funny with forwarded headers and for some reason, you have to add middleware that is found in the Microsoft.AspNetCore.HttpOverrides nuget package. Once this is installed, you need to do the following in Configure:
    3. var options = new ForwardedHeadersOptions
    {
        ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto,
        ForwardLimit = 2,
    };
    options.KnownNetworks.Clear();
    options.KnownProxies.Clear();
    app.UseForwardedHeaders(options);

  3. You can then get the client ip address from Request.HttpContext.Connection.RemoteIpAddress
  4. Note that your ingress will probably use combined ipv4/6 ports so you might see the request ip address as something like ::ffff:10.11.12.1 so if you want something normal, you can call MapToIPv4() on the RemoteIpAddress.
  5. Now with respect to secrets, since the Docker build is pre-packaged, you cannot simply use different appsettings files without them all being present in the build which means production secrets in the code. Instead, using the instructions from here, I decided to inject the production app settings using a Kubernetes secret.
  6. Firstly, you need to create your actual file like appsettings.secrets.json locally. Then create it as a generic secret into Kubernetes using e.g. kubectl create secret generic secret-appsettings-myservice-production --from-file=./appsettings.secrets.json
  7. Then modify your deployment to include an environment variable for ASPNETCORE_ENVIRONMENT (which might or might not be needed but is generally a good idea).
  8. Also, you will need to specify a volume to point at your secret inside your deployment. The name can be whatever, the type will be secret and the secret name as you just created.
  9. Once the volume is specified, you also need to mount the volume into your app container so that your config code can find it. In the example, we mount it at /app/secrets, which is suitable for Linux. Not sure what we would use for Windows. You can set the mount to readonly.
  10. You now need to modify your dotnetcore app to look for the secrets, if present, and override the default config. You might have seen this before and it sometimes appears in the Startup class but in my case (dotnet core 2.2) it was using the default builder, which means the syntax is slightly different - it already looks for the normal config settings. The following code shows how CreateWebHostBuilder looks now:

    public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
        WebHost.CreateDefaultBuilder(args)
            .ConfigureAppConfiguration((hostingContext, config) =>
            {
                config.SetBasePath(hostingContext.HostingEnvironment.ContentRootPath);
                config.AddJsonFile("secrets/appsettings.secrets.json", optional: true);
            })
            .UseStartup();
Easy eh?