One of the main benefits of Docker and a feature that makes it usable in the first place is the use of cacheable layers.

In short, most statements in a docker file make whatever changes they make at which point the differences from the previous filesystem are tar'd and then the hash of this is used to store a copy of just that layer.

When we build again, if the files are the same then Docker can determine that it doesn't need to run the statement(s) again but can simply use the cached layer from before.

This is the main reason why you need to keep rarely changing things as far up the dockerfile and more changed items lower down.

However, it doesn't always seem to work and sometimes it doesn't seem to use the cache even though you don't think you have changed anything that is relevant.

Here are some reasons:

  1. Do not use COPY . . as a way of reducing the number of lines you need to do the copy. In most cases, you will only need a few files to begin with that you might, for example. run npm install against. You might then copy the rest of the source files to build it. If you COPY . . before running npm install, then any time you change ANY source file, you will run npm install all again!
  2. Make sure you have setup a relevant .dockerignore file to not send anything that is not required up to the Docker engine. If you don't have the file and it is uploading intermediate or output files from your local (non-Docker) build, then any time these files change, you can again trigger stages to run again.
  3. Make sure that your docker files are ignored in .dockerignore. This is especially true if you have non-standard names for them since they won't be ignored by default whereas "dockerfile" might be.
  4. In some cases you just need to slow down and think about what is happening and why it is not using cache. Since you can see the first stage that is not using cache, it should be fairly simple to look at what is being copied at that stage and why it might have changed. When we were first using a build server, we were injecting the build version before compiling. When the build ran a second time, git noticed the changed file and reset it back before having a different version injected. Docker correctly saw the change and rebuilt the entire app. We changed it so that version injection only happens before publish and not at the beginning. This made a big difference in rebuild times.