Background
I try to iterate a huge pandas dataframe (5GB in gzip parquet format) and before I iterate it, I have extracted the useful info from the df and converted that part to a list. It is list of list the length of which is about 4M and the length of each element in which is about 100.
Problem
When I iterate the huge list, it will stop at some points ,like 1Mth, for about 13s and then continue executing the following iterations and then at some points like 2Mth, stops for 13s and then continue. For each iteration, the execution is very fast and it should not stop at some points for about 13s.
My debug
- I use VSCode to debug it and find that each time the stopped point is different, but, the stopped points are very close, for instance, the first time I run the code, it stops at 521057th iteration for 13s, and the second time, it stops at 521143th iteration.
- When I find the progress bar stops which means the code arrives at the stop point, I set break point on every line within the for loop and find all the stop points are not triggered. And after about 13s, the code trigger the line below the
for. It seems that the it stops at theforline, may be wait to get data?
OS
- GCE VM
n1-highmem-96. - The memory is about 680GB.
The code takes about 20% of the memory when it is running.