I can't figure out a memory error I get when running a program that parallelizes the reading of XMLs hosted online by Amazon containing 990 tax return data on my Windows machine running Windows 7 64-bit.
I iterate through an index of IDs from a CSV downloaded locally on my machine to store the link to each XML (there are around 200,000, roughly):
URLs = 'https://s3.amazonaws.com/irs-form-990/' + df990Orgs['OBJECT_ID'].astype(str) + '_public.xml'
and then I use a parallelized for-loop to pull all the data with an XML parser I wrote and return a dictionary of all the fields and their values (roughly 200 fields each for 200,000 records):
ListRows = Parallel(n_jobs = -1)(delayed(XMLParsefromURL)(URL) for URL in URLs)
Then, I create a pandas dataframe from ListRows and save that as a CSV:
df = pd.DataFrame(ListRows)
df.to_csv('OrgInfo' + strname + '.csv')
If I only run the parallelized loop for 10,000 elements, it runs perfectly and I get a populated CSV. However, if I let it run for all entries (~200,000), I inevitably get the following error mid-run:
Exception in thread Thread-3:
Traceback (most recent call last):
File "C:\Users\jnimgaonkar\AppData\Local\Programs\Python\Python36-32\lib\threading.py", line 916, in _bootstrap_inner
File "C:\Users\jnimgaonkar\AppData\Local\Programs\Python\Python36-32\lib\threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\jnimgaonkar\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\pool.py", line 463, in _handle_results
task = get()
File "C:\Users\jnimgaonkar\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\connection.py", line 250, in recv
File "C:\Users\jnimgaonkar\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\connection.py", line 318, in _recv_bytes
return self._get_more_data(ov, maxsize)
File "C:\Users\jnimgaonkar\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\connection.py", line 344, in _get_more_data
f.write(ov.getbuffer())
MemoryError
Curiously enough, I don't get any errors when running it on my macbook air (which has only 4 gigabytes of RAM) - the whole thing compiles perfectly and I get a CSV with the correct number of records on my machine.
From my research in trying to figure out this problem, I know it might have something to do with Windows not being able to fork() (or at least not making it convenient), but I don't know enough to make use of that information to find a solution. I would appreciate any help you can give me. Thank you.