How to write to a single csv from multiprocessing without losing data in python?

Viewed 14

I have a multiprocessing process running pool that runs a function , where each functions writes to a single csv. What I am noticing is that some of the data is not being written into the csv even though the function runs properly. Is there a way to handle this or must I create different csv files for each pool?

Here's a snippet of my code:

def telephonfunction(q1,q2):
    #path to write output to
    path='newnumber.csv'

    oldnumbers=pd.read_csv('tel.csv') 
    for x in oldnumbers['msisdn'][q1:q2]:
      #run some action here
      #output
      result.to_csv(path, mode='a', index=False, header=False)
      print('x:success')

#gets the position to be passed into the function
tel=pd.read_csv('tel.csv')
length=len(tel['msisdn'].tolist())
q1 = int(length/7)
q2 = int(length/7*2)
q3 = int(length/7*3)
q4 = int(length/7*4)
q5 = int(length/7*5)
q6 = int(length/7*6)

data=[
    [0,q1],[q1,q2],[q2,q3],[q3,q4],[q4,q5],[q6,length]
]
    
if __name__ == '__main__':
    p = multiprocessing.Pool(6)
    p.starmap(telephonfunction,data )

I checked the log and checked the path that's it's being write into and noticed that some results are not being appended.

0 Answers
Related