We are having very large data files, for example assume a 3D volume with 2048x2048 matrix size with a slice depth of 20000.
Originally, I had my own central memory management where each data is backed with a "page-file" on the disk. During processing I track the process memory and if the memory is low I check which slices haven't been touched for a while and page them in my own hand-made paging-file. This results of course in some form a zig-zag when you look at the process memory, but this methods works even if the total size of my files is much larger than the available RAM. Of course putting more RAM inside improves the situation but the system is able to adapt to these situation. So if you pay more you get more speed ;-)
But, the method is of course hand-made, I need a separate thread that watches over the allocated files and data, I need special methods that handle the paging etc.
So I decided to have a look at memory mapped files...
When I now create a memory mapped file (Read&Write) and run through the data slice by slice and get access to the data with a ReadOnlySpan<> in a for loop the whole memory is consumed and you see the process memory lineary growing until the whole memory is consumed. Afterwards the system goes into swapping without coming out ever again. Its even so that the system freezes and only a restart helps to recover.
So obviously the MM-Files are not balancing themselves and when it comes to the point where we a reaching the RAM maximum, the system is exhausted and is obviously helpless.
The question is can I control when the system does a swapping and advise which regions can be swapped? Or do you have suggestions for better approaches?
I have created a example that does following
- It creates a Volume as input
- It creates one or many processor that simply add 10 to each slice in each processor.
- Since everything is created only on access, a Parallel.For iterates over the slices and requests the result slice. Depending on the amount N of processors in the chain each slice of the result should have a value of (N+1)*10.
See here: Example Code
Update [26.09.2021]: My latest investigations have shown that there are two independent process running. One process (my process) is writing or reading data into the memory mapped file(s). Anothter system process is trying to manage the memory. While the memory is growing the system is obviously very lazy flushing data really to the backing memory mapped file or the paging file.
This leads to a bad condition. When the maximum RAM is reached, the memory manager starts to empty the working set and flush the data to disk. While the system flushes, I'm still producing data which fill the memory again. So in the end I'm coming to a point where I'm always using the maximum RAM and never come out.
I tried a few things:
- VirtualUnlock enables me mark regions as pageable for the memory manager.
- You can call a Flush on the ViewAccessor.
- Since the workingset is still showing high memory usage, I call EmptyWorkingSet to empty the working set, which immediately returns.
What I need in the end is something to see if the memory manager is currently unmapping so that I can slow down the process.
I have to search more...