One of the common principles of security is not to include production credentials in development where they are easily shared, lost or exposed to hackers. In the world of DevOps and Infrastructure-as-code, it is even more important where deployments are automated, to make sure that credentials are locked down suitably.

In the world of Microservices and Kuberenetes, there are generally two types of data that are environment-related. One is simply environment variables that are not necessarily secret but are required to perhaps set a connection string or the level of logging. These can, and usually should, be set as environment variables in your deployment. They can easily be substituted using CI/CD tools for each environment.

Secrets, on the other hand, could be added as environment variables but are not necessarily completely environment-dependent and are also easier to expose to any systems running on that node. A better solution is to use Kubernetes secrets, which are really just data that are mounted as files into the pod containers and can therefore be read securely.

The need for automated Secret deployment

We had a scenario the other day where we realised at very short notice that the secrets for our dotnet core microservices were expired and there are a number of problems with this. Firstly, I only had scripts to update the secrets because it is not something I do every day and hadn't automated it. Secondly, the file watcher used by json config files in dotnet core does not see file changes to these secrets so even if you update them, you either have to redeploy, restart or scale down and up the pods to get new instances that will reread the secrets. We had issues for around 15 minutes but fortunately, they are not critical infrastructure for most parts of our system and it was 5:30pm!

I decided I needed to do something more automated so that it would be easy to get a quick deployment, ideally using the existing variable substitution that we use with most apps via Octopus Deploy.

Setting up the Dotnet Core microservices

Initially, I used the same name for the secrets file that was injected into the pod and read with AddJsonFile in the service. This required a lot of fiddling around when deploying changes since you have to use the correctly named file as the source to kubectl create secret. All I did was add an additional optional file load with a service-specific name (system, auth, templates etc.) so that it wasn't a breaking change. 

I also added a ps1 file which does the magic. It basically deletes each of the app settings secrets and then creates a new version based on the local json files.

kubectl delete secret secret-appsettings-auth-production --ignore-not-found=true
kubectl create secret generic secret-appsettings-auth-production --from-file=./appsettings.secrets-auth-service.json

As simple as that. Octopus can run this script directly from a package.

As with all development, especially with production systems, I tested it at each stage with test names to make sure the syntax was correct and all worked as expected.

Configuring Team City

Our build server then needed to create a package containing all the source json files for variable substitution as well as the ps1 script. In Team City, this is a single step build configuration using the OctopusDeploy: Push Packages step which you can set a bunch of wildcards all pointing to the same zip file including the build number and then ticking the box to "publish packages as build artifacts", this was helpful for testing, although it was not needed in general.

I ran the build once and checked that the zip file artifact only contained the various json files and the ps1 deployment script. Since I was not automatically invoking the Octopus deployment step, this was safe to do.

Configuring Octopus

Fortunately it wasn't too bad to setup Octopus. I couldn't get Octopus' deploy secrets step to work since it looks like it only works with key/value pairs and not file-based json. I instead used the Run Kubectl CLI script step. In some ways it is just a script but the functionality allows the system to work out which environment to login with before running the script so that kubectl hits the correct location.

Inside this step, I simply specified that the script would run from a package and pointed to the package that Team City was pushing. I also locked it down to Production because our dev environment works differently and doesn't use secrets.

I included the Json Configuration Variables and Substitute Variables in Files features so that Octopus would do the variable substitution and also added the relevant parameters to an Octopus variable set.

When I initially tested it, I only included the script for one of my test secrets. After deploying and using kubectl, I downloaded the secret, base64 decoded it and confirmed it had done both types of substitution.

Now all I need to do in emergency is update the keys in Octopus, hit the deploy button and either deploy the microservices or manually scale them, which only takes seconds. If we are more organised, we would be updating secrets pre-emptively and also enabling scheduled microservice deployments to ensure we are getting OS fixes.