I'm trying to find a way to trigger a memory dump when a container experiences an OOM exception. The problem is that when this happens, the container is restarted so we're unable to get a valid dump. I saw a similar article outlining how to do it in Java, but I'm not sure if/how this can be translated to .NET Core.
2 Answers
You have 2 ways to do this.
For both ways to work, .Net Core needs to support a flag which tells it to dump on crash and a flag to pass it the path of where to dump.
Have a sidecar with shared storage on the microservice pod. In the sidecar, have a script that monitors the storage and copies the files dumped there to a location like a s3. I guess this is the one mentioned in the Java link
Have a storage common storage attached to all the pods of a Microservice, something like a NFS, in amazon's case, EFS. Use that efs in your deployment and make the .Net dump to that storage. Then it's only a matter to getting the dump out which you can do via a debug pod or a jumphost attached to the EFS
You can mount a shared storage, and set the according Environment variables to tell .net core write dump to the shared storage when its process crashes.
More details are in this document https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dumps#collect-dumps-on-crash.
Related