What is causing the large memory commit upon importing pandas

user_3538468 0
Viewed 147

I have been working on this issue for hours and have not come across anyone else with this issue. I have recently switched from Linux to Windows 10 and I am running into memory issues with my python applications.

All my python applications have taken up ~450MB of Memory Commit. This is becoming a problem as we are deploying multiple instances & new applications. Upon inspection of the problem, pandas seems to be the cause. After stripping out most of my applications, I can recreate the issue using simply:

  • Creating a venv environment with Python 3.8.8 (also tried 3.8.3 and 3.8.5)
  • installing pandas
  • calling python within the environment
  • running import pandas

Screenshot of the memory usage. The newly created process is 291724

Does anybody know what is causing this issue, and how it can be prevented? Surely importing pandas should not reserve so much memory, nor should it keep this amount reserved after finishing an import?

Update 06/06 Here is another screenshot as requested by anon01. The commit of the python process is very large compared to the Working Set, Shareable and Private memory.

Screenshot of the memory usage sorted by Commit as requested

pandas memory python
1 Answers

I'm afraid I wasn't able to solve this issue, but some things did help. Foremost, downgrading to Python 3.7 did reduce the memory commit by >30%. Switching to a different version of pandas also helped reduce ~10%. I guess just trying out some older/newer versions is the best approach to the problem.

By refactoring some software, I was able to remove the pandas dependency in some applications. Not sure if this answer is satisfactory, but I hope it gives people who come here in the future some pointers to what works.

user_35384680
Related