I tried to got 6 million lines from 70 million lines each time and repeated this process for 10000 times. My goal is to put this 60 Billion lines in one test file ,so that I can count some specific words' frequency for 10000 times.I wrote a python script and used the cluster to run the script.
Here is my code:
import numpy as np
write_fname=open('/etc/file_to_write.txt', 'w', newline='')
f = open('/etc/file_to_sample'.txt', 'r')
f_l = f.readlines()
for i in range(10000):
Random_data_line= np.random.choice(f_l, 6165307, replace=True)
tmp_data = str(Random_data_line)
write_fname.write(tmp_data)
write_fname.close()
and the cluster later report this error in the log:
Traceback (most recent call last):
File "/etc/user/etc/script/sample.py"line 6, in Random_data_line= np.random.choice(f_l, 6165307, replace=True)
File "mtrand.pyx", line 894, in numpy.random.mtrand.RandomState.choice
MemoryError: Unable to allocate 875. GiB for an array with shape (78267140,) and data type <U3002
I knew that python has GIL, so it only can fully use one core, is this related to my issue? How should I do to solve it?
(I also have tried to split my source data into 3 file, but it seems still too big for the memory space.)
I also wonder if I sample one time and count words and repeat this "sample-counts words" process more accessible ?