DbContext memory leak
DbContext in EntityFramework is a double-edged sword. It seems to make things easy but it is also easy to misuse, especially in dependency injection. Entity Framework itself is also naughty because of the way it magically starts up if it finds configuration rather than needing to be explicitly initialized.
Firstly, we got the app to leak data in production and then did a memory dump using adplus, but I don't know if right-clicking the task in task manager would produce the same output. This gave us the entire 2.5GB memory allocation.
Download the dump to my local machine, install windbg from the Windows SDK and opened the crash dump (File -> Open Crash Dump). This simply loads the memory contents and gives you the chance to load and execute various commands. Firstly to load the sos module using .loadby sos clr and then !dumpheap -stat, which shows a large table of allocated objects, the total count and the total size.
It orders it by total size which is usually OK for memory leaks so I copied the bottom 50 rows, pasted it into a text file and imported into Excel as delimited text so I could play with the numbers and work them out in MB etc (I could possibly have done this in windbg!). I realised that the largest allocation by far was System.Data.Entity.Core.Objects.StateManagerValue and it didn't take much to realise this was related to EntityFramework. I then went back to my list and searched for DbContext and found 5307 DbContexts still allocated. This might have been pre-GC but in this case, I noticed that the type we wrap DbContext in only had 413 allocations and we don't allocated DbContext anywhere else.
To dig further, you can inspect a single type of object and see each individual allocation for that type using !dumpheap -mt 0006df77 or whatever address is listed next to the object you want to inspect. Once you have this list, choose an item earlier in the list (which is likely to be leaked and referenced) and then you can run !gcroot 242443ec using the address of the item you have chosen. This should give you a stack trace of the references that are causing your allocation to avoid GC. In my case, it traced back to LazyInternalContext, which didn't tell me much more than I already suspected. If it says there are no roots then it means the item is eligible for garbage collection, it just hadn't happened yet when you took the memory dump.
I didn't know whether I could track allocations of this internal MS class to find out what was happening but fortunately, I had recently added some EntityFrameworkCore nuget packages to try and fix a strange owin problem so I assumed (correctly) that these were doing unexpected things even though we weren't directly using them.
We simply backed them and tried the app again and it was fine!
Some basics to avoid memory leaks
- A DbContext is a lightweight object so you should use it and dispose of it. In most cases, you can do this inside a using statement. Do not assume you will benefit by keeping it around, in most cases you will get increased memory usage (states are remembered by default) but you can also leak memory.
- If using IoC, it should be registered as Transient (the default in most cases) so a new one is created each time you use it. If using the repository pattern, ensure that either your repos are also Transient or inject a Transient resolver into a Singleton so that a new instance is created each time inside the repo.
- Do not install EntityFrameworkCore alongside EntityFramework, this caused a massive memory leak in an app we host that only has a handful of staff using it but quickly reached several GB of RAM allocation (more details below). This is because of the automatic way it adds itself to your DB pipeline without you explicitly calling anything.
- If you are building a new app, start with the basic code and test it for memory leaks before deciding that your pattern is correct and you can then duplicate it. We had challenges with injecting Dapper connections along side DbContext for tests etc. that we had to work out as we went along.
Detecting the memory leak
I've always been impressed by the people who can get really down and dirty to debug things but I got to do it for our leaking application.Firstly, we got the app to leak data in production and then did a memory dump using adplus, but I don't know if right-clicking the task in task manager would produce the same output. This gave us the entire 2.5GB memory allocation.
Download the dump to my local machine, install windbg from the Windows SDK and opened the crash dump (File -> Open Crash Dump). This simply loads the memory contents and gives you the chance to load and execute various commands. Firstly to load the sos module using .loadby sos clr and then !dumpheap -stat, which shows a large table of allocated objects, the total count and the total size.
It orders it by total size which is usually OK for memory leaks so I copied the bottom 50 rows, pasted it into a text file and imported into Excel as delimited text so I could play with the numbers and work them out in MB etc (I could possibly have done this in windbg!). I realised that the largest allocation by far was System.Data.Entity.Core.Objects.StateManagerValue and it didn't take much to realise this was related to EntityFramework. I then went back to my list and searched for DbContext and found 5307 DbContexts still allocated. This might have been pre-GC but in this case, I noticed that the type we wrap DbContext in only had 413 allocations and we don't allocated DbContext anywhere else.
To dig further, you can inspect a single type of object and see each individual allocation for that type using !dumpheap -mt 0006df77 or whatever address is listed next to the object you want to inspect. Once you have this list, choose an item earlier in the list (which is likely to be leaked and referenced) and then you can run !gcroot 242443ec using the address of the item you have chosen. This should give you a stack trace of the references that are causing your allocation to avoid GC. In my case, it traced back to LazyInternalContext, which didn't tell me much more than I already suspected. If it says there are no roots then it means the item is eligible for garbage collection, it just hadn't happened yet when you took the memory dump.
I didn't know whether I could track allocations of this internal MS class to find out what was happening but fortunately, I had recently added some EntityFrameworkCore nuget packages to try and fix a strange owin problem so I assumed (correctly) that these were doing unexpected things even though we weren't directly using them.
We simply backed them and tried the app again and it was fine!