.NET Core application running in docker gets OOMKilled if swapping is disabled #851
Comments
output of the test toolOutput of docker inspect |
devKlausS
changed the title
|
@devKlausS thank you very much for the detailed analysis and repro details! I will look into it. |
|
@devKlausS can you please tell me what Linux distro is the one you are using? I have tried to repro it on my Ubuntu 16.04 and the behavior is different. My container doesn't get OOM killed, but somehow hangs instead. I cannot stop it or kill it (the docker stop / kill commands just run for a while and then terminate with no message, but the container keeps running). And this is the last line of output before it hung: |
|
@janvorli I'm working together with Klaus on this issue. It seems like your OOM killer is disabled actually. Kernel cgroup documentation states that malloc call will hang (sleep) if OOM killer is disabled (which isn't very useful, either). It is however not disabled by default, and I only know how to disable it for a single container. Kernel docs reg. OOM killer: |
|
This is similar to what I observed in dotnet/coreclr#16906. Docker does not respect the memory limit, so the GC is triggered too late. If swap is enabled it does not matter if the memory limit is violated, as the excess memory is just swapped out. |
|
@janvorli I'd like to add that to us the root cause seems to be that the memory limit, albeit implemented by @rahku in dotnet/coreclr#10064 is not read correctly by cgroups.cpp @hpbieker we didn't find your issue actually although we were searching a lot :) |
|
@dmpriso Ah, that is likely the reason for the hang. I've completely forgotten about that behavior, but remembered now that you've mentioned that. |
|
Great, I've enabled the oom killer and disabled the swap and now I can repro it. |
|
While I can repro the issue, in my case, the limit is read correctly from the cgroups files and the path is also correct: (gdb) p mem_limit_filename (gdb) p physical_memory_limit So the fact that in your case, it was trying to use a wrong path most likely means that on your host, the Could you please run /bin/bash in your container (add cat /proc/self/mountinfo | grep "cgroup"The fact that we still get killed due to the OOM on my machine even though the limit is read correctly could be due to the fact that the GC is not the only part of coreclr that consumes memory. There is also memory consumed by native allocations in the runtime, which includes the JIT, the managed assemblies, the native binaries etc. Also, we are still missing low memory notification for GC on Unix (https://github.com/dotnet/coreclr/issues/5551), but even that would not prevent the OOM kill to happen in all cases (especially when memory is being allocated quickly). |
|
This is on dotnet_mem_allocator as per your instructions This is on ubuntu:latest where we debugged cgroups.cpp Our cgroup_test tool also fails with the .NET core image: cgroup_test just consists of cgroup.cpp with a simple main function: Regarding the native allocations done by the runtime: Isn't code allocating unmanaged memory supposed to call |
|
@janvorli could you spot any difference in our and your mountinfo output? |
|
@dmpriso this is weird - your dump matches my one and yet it gives me the correct limit. However, now I've realized something - you are testing code from https://github.com/dotnet/coreclr/blob/master/src/gc/unix/cgroup.cpp, which is used by the standalone GC (if you use it). I was debugging the copy of this code that's in PAL and is used by the GC embedded in the coreclr - https://github.com/dotnet/coreclr/blob/master/src/pal/src/misc/cgroup.cpp. While these should match (except for differences due to one using PAL types / functions and headers and the other using standard ones, there might be a subtle bug causing the malfunction of the one that you've tested in your little c++ test.
User code - yes. But even with user code, it would be complicated if you use code that you don't control - e.g. 3rd party libraries. While you could possibly install hooks for the malloc / free calls, you cannot track mmap that some libraries can use. And even with mmap, it would be difficult to track how much physical memory it has allocated, since it can map memory in "lazy" manner when the physical pages are allocated at the first access to a memory page.
The purpose of this method is to accumulate the allocated bytes and trigger a GC if we go over a threshold that we dynamically update each time we cross it. The idea is that you'd use it before doing native allocation and GC can free some memory if it seems it may be needed. But the accumulated pressure is not used by the GC itself. |
|
Hmm, I've tried the same thing as you did - just compiling the https://github.com/dotnet/coreclr/blob/master/src/gc/unix/cgroup.cpp with your bit of code to test it added. And it still worked :-(. |
|
@janvorli I feel sorry, while digging into that issue I must have picked up the file from the original commit b2b4ea2. Your snipped as well as the current version correctly read the limit. My fault! So while my analysis is wrong, the issue still exists but for the reason you mentioned. Do you have any idea how to work around that? |
|
One feature that is not implemented for Unix yet is the low memory notification (#5551). That would fix the issue in case the memory allocation doesn't happen with an extremely high rate that would prevent the notification to have the desired effect. What I mean is that if you allocate large blocks of memory in a tight loop and deplete the available memory in a fraction of second, the OOM might happen in the interval between the regular polls of available physical memory. So one poll would find there is still a plenty of memory, but we OOM before the next regular poll comes in. |
|
What would be required for a 100% solution, on top of that? |
|
I tested with release .NET CORE 2.1.5 but unfortunately the issue still exists. I created new docker images based on 2.1.5 SDK and performed the tests like described above. Both test images get OOMKilled ServerGC=false alloc/free .NET CORE 2.1.5
ServerGC=true alloc/free .NET CORE 2.1.5
|
|
Based on what @janvorli said, it is expected this test program still runs out of memory: the rate of allocation is high compared to the provisioned memory / poll interval. Do your production applications run OOM with 2.1.5? |
|
@devKlausS I'll try to debug it locally again to see if I can pinpoint the issue you are hitting. |
|
@devKlausS please try: I have also tried this, after this memory utlization has come down to more than half. btw I am running .net core in linux - docker container - aws ECS related article: https://blog.markvincze.com/troubleshooting-high-memory-usage-with-asp-net-core-on-kubernetes/ |
|
this is related to other long running (closed) problem as well (dotnet/dotnet-docker#220), actually java has solved to listen to the cgroup and not on host machin/vm memory (or using lxcfs it works as intended), but .net somehow doesn't use /proc provided infos to use in decisions on high memory pressure. So I would preffer the environment var or dotnet run cli option, which in this way can be used eith in e.g. ecs or kubernetes deployment descriptor with helm chart, where we can substitude mem resource limit and this option with the same value. |
.NET Core uses cgroup limits. Available memory is determined by polling the proc file system. Because of the polling interval, it is possible to allocate so much you get yourself killed, before .NET Core sees it should perform a GC. That is what happens with the code in the top comment. |
Is the polling used to check For now I am manually invoking garbage collection on a timer, but it seems fairly flaky. |
|
@kriskalish Might be relevant for your question dotnet/coreclr@a25682cdcf @tmds In your opinion, which is the best suited approach then? As far as I can understand, setting Workstation GC looks like to be the "safest" option to constrain memory consumption, but it may become impacting on throughput-sensitive scenarios. Or is there a reliable way to prevent OOM kills while still preserving memory w/ Server GC (w.r.t. your point on cgroup polling time)? |
|
@tmds cgroup and proc is not the same. So what is used by netcore? proc only reflects cgroup if it is mounted by lxcfs or some other mean of fuse. In our microservice containers only one netcore process runs, so it doesn't race cgroup limit with other processes, however it continuously oom kill itself. I've also tested 2.0 with just memory pressure setting meminof under proc to a relatively low value, and it gets oom killed. Investigating that time the heap manager, as I remember it used kernel call to get phisical mem, not the proc file system. Can you tell me which version/release of netcore switched to poll proc? |
This blog post series will give us some more guidelines.
Switching to workstation GC will help. GCs will be triggered more soon because there is a single, small gen0+1 segment.
.NET Core reads the cgroup limits and usage from the proc filesystem. This code is in the cgroup.cpp files in this repo.
You need to run at least 2.1.5. |
|
Hello i am having the same kind of issue on asp net core 2.2, on a docker (ubuntu) on AWS ECS. I tried to run the docker container on my machine and it does not kill itself. |
|
this is the one I wanted to ask you about, @janvorli - a while ago you mentioned you still wanted to do some work for this so just checking. |
|
@saixiaohui have you tried 3.0? |
|
Having the same issue. Will upgrading to 3.0 fix this issue? |
|
As a workaround this issue can be solved by running |
|
Upgrade from 2.2 to 3.0 and the same problem occurs. With 20 requests, concurrency level of 2..
At 2.2, it was 2 clicks at the same time and it died ... now it's getting longer alive before that. |
Dotnet-GitSync-Bot
added
the
untriaged
|
Same question as #852 (comment) ... is this issue still a problem with 3.x. Info on 3.x for containers: https://devblogs.microsoft.com/dotnet/using-net-and-docker-together-dockercon-2019-update/ |
jeffschwMSFT
removed
the
untriaged
|
We face the same issue. Switching from .NET core 2.2 to 3.1 made it even worse for our application. |
|
The issue is still relevant for netcore We facing the problem while migrating an asp.net application with high memory pressure between two sets of servers. We use docker swarm mode to orchestrate services and cgroups for a memory limits. Exact same image was used to start a container on both servers sets and it fails with out-of-memory on a new ones. The servers were configured with ansible scripts, so they use the exact same configs, docker engine version and both has disabled swapping. Only differences between the old servers and the new servers were an OS version and different environment type. So, attempt to run a container with memory limit leads to Attempt to run a container without memory limit leads to a "half-dead-working" application with really high memory consumption (4x above normal), without ability to process requests (with connection refused). One of my hypothesis was about different ubuntu versions. To validate it I deployed stage copy of production node (with Ubuntu 18.04.4 on a virtualized machine) and made the application with production setup run. The run was successful. So maybe problem with GC is somehow relevant to a system run mode? Temp solution, that make it work: explicit use of <PropertyGroup>
<ServerGarbageCollection>false</ServerGarbageCollection>
</PropertyGroup> |
|
Whoa, hope this could be helpful for some researchers. After further investigations I've noticed that there is big difference between my servers in amount of available logical CPUs count (80 vs 16). After some googling I came across this topic dotnet/runtime#622 that leads me to an experiments with CPU/GC/Threads settings. I was using The only thing that make it work with Server GC is What is interesting, previously I have mirgated 3 other backend services to a new servers cluster and they all go well with a default settings. Their memory limit is set to It keeps working in any range between Of course, it's relevant only for my particular workload, but may be someone could see any relations. |
|
Hi, is there a plan to fix this sooner ? We are also hitting this problem |
|
I have finally tested this with .NET 6 preview 6. We have made some changes to how we detect memory load on Linux recently, so I wanted to give this a try. It seems that both the client and server cases now behave correctly: The total memory keeps oscillating between 40 and 60 Mb even if I let it run for a long time. Server GC version behaves the same way: |
|
@devKlausS these were tested with swap disabled. Do the result look good to you so that we can close this issue? |
|
@janvorli I finally tested it with .NET 6 preview 7 in our k8s cluster and it looks good. Thanks for your help! |
ghost
locked as resolved and limited conversation to collaborators
and others
We are running .NET Core applications inside docker containers on a kubernetes cluster. Docker containers get OOMKilled because GC is not executed.
To test this we created a test program which allocates arrays in a loop. These arrays should get deleted by the GC. The program is executed in a docker container with 100Mb memory limit. Kubernetes is not involved in the tests.
When running the test with swap enabled everything works as expected and the GC gets triggered when the total memory reaches 100Mb. The test never gets killed.
When running the test with swap disabled the test gets OOMKilled when the total memory reaches 100Mb. We have tested this behaviour with ServerGC=true|false and with .NET Core 2.0 and 2.1
Enabling swap is not an option cause it's neither recommended nor supported by Kubernetes.
Code
you can find out test program here: https://github.com/devKlausS/dotnet_mem_allocator
Docker images
ServerGC=false alloc/free .NET CORE 2.0
ServerGC=true alloc/free .NET CORE 2.0
ServerGC=false alloc/free .NET CORE 2.1
ServerGC=true alloc/free .NET CORE 2.1
cgroup limits
After reading multiple issues according to .NET Core GC and docker limits we started to dig into the cgroup limits. We compiled https://github.com/dotnet/coreclr/blob/master/src/gc/unix/cgroup.cpp and executed the program inside the docker container. As shown in the following screenshot the the path of the memory.limit_in_bytes file is pointing to the directory of the host machine and the file open operation fails.
The next screenshot shows the volume mounts of the container. The host machine path is mounted to /sys/fs/memory inside the container. When reading memory.limit_in_bytes we see ~300MB (we pass -m 300M to the docker container)
It seems like CLR is not able to read the physical memory limits from cgroup and therefore the GC is not triggered correctly. As a result the process is killed because of OOM. We guess that CLR is triggering GC when it has to do swapping and this is the reason why our process is not killed when swapping is enabled.
The text was updated successfully, but these errors were encountered: