Can you force python to use specific parts of memory

Quasi 0
Viewed 27

So I have some pretty large files which I want to store in my memory. The problem is that as soon as the system memory fills, everything slows down.

Since windows has the possibility to use normal memory as virtual system memory, I wanted to ask if it is possible to force a python environment to use this virtual memory instead of the normal system memory.

memory windows python
1 Answers

All Windows applications use virtual memory. Virtual memory has some advantages, like

  • the addresses in each process start at 0 and are contiguous
  • pointers only point within your process, they can't point to other processes. This is security relevant

Virtual memory is managed in pages à 4 kB (for simplicity, let's not consider edge cases). Each of these pages can be

  • in RAM (fast access)
  • in the page file (slow due to disk; notice "page" here)
  • non-existent

Only the operating system itself and kernel drivers will have access to physical RAM and thus have access to the memory of all applications. This is not what you do with Python.

In Windows, there is no such term as "normal memory" or "normal system memory".

As far as I know, Python uses obmalloc() to allocate memory. obmalloc() uses the C malloc() under the hoods. malloc() again uses virtual memory from the operating system.

So: Python already uses the virtual memory of Windows.

If you want to know more details about the virtual memory of a process, use SysInternals VMMap. It's free.

If you want to know the details on how Windows deals with memory, read the book "Windows Internals" by Mark Russinovich (or maybe sometime in the future by Pavel Yosifovich). Memory is covered in part 1 of volume 7.

Thomas Weller0
Related