I have a multiprocessing process running pool that runs a function , where each functions writes to a single csv. What I am noticing is that some of the data is not being written into the csv even though the function runs properly. Is there a way to handle this or must I create different csv files for each pool?
Here's a snippet of my code:
def telephonfunction(q1,q2):
#path to write output to
path='newnumber.csv'
oldnumbers=pd.read_csv('tel.csv')
for x in oldnumbers['msisdn'][q1:q2]:
#run some action here
#output
result.to_csv(path, mode='a', index=False, header=False)
print('x:success')
#gets the position to be passed into the function
tel=pd.read_csv('tel.csv')
length=len(tel['msisdn'].tolist())
q1 = int(length/7)
q2 = int(length/7*2)
q3 = int(length/7*3)
q4 = int(length/7*4)
q5 = int(length/7*5)
q6 = int(length/7*6)
data=[
[0,q1],[q1,q2],[q2,q3],[q3,q4],[q4,q5],[q6,length]
]
if __name__ == '__main__':
p = multiprocessing.Pool(6)
p.starmap(telephonfunction,data )
I checked the log and checked the path that's it's being write into and noticed that some results are not being appended.