I am trying to process the number of csv files using multiprocessing.
Below is the code -
UPLOAD_FOLDER = Path("/home/Documents/Upload")
def getDate(a, b):
return some_date
def post_process(filepath):
print(f'Process {os.getpid()} working file {filepath}')
source_df = pd.read_csv(filepath, sep=';', index_col=None)
source_df = source_df.dropna().reset_index(drop=True)
if len(source_df) > 0:
/* some processing code */
filename=re.split(r'[/]', filepath)[-1]
source_df.to_csv(UPLOAD_FOLDER/f"{filename}.csv", sep=';', mode='a', index=False, header=False)
print(f'Process {os.getpid()} done working file {locofile}')
def parse():
pool = multiprocessing.Pool(processes=2)
result = pool.map(post_process, glob.iglob('/home/Documents/Docs/**/*.csv'))
if __name__ == '__main__':
parse()
What I am expecting is to read all the csv files from multiple folders, process them and store to different location as csv. If I am running with 2 processes, usually every process should read, process and store the files to the location then pick the next file. But it is starting the process for multiple files with more than two PID's, and after sometime the process is getting stuck.
Below are the logs -
Process 330987 working file--> file1.csv
Process 330985 working file--> file2.csv
Process 330987 done working file--> file1.csv
Process 330987 working file--> file3.csv
Process 331198 working file--> file4.csv
Process 331198 working file--> file5.csv
Process 330985 done working file--> file2.csv
Process 330985 working file--> file6.csv
Process 331710 working file--> file7.csv
Process 331858 working file--> file8.csv
Process 331858 done working file--> file8.csv
Process 331858 working file--> file9.csv
Process 331858 done working file--> file9.csv
In logs if you see there are more than two PID's running. Please help me to find the solution OR any better approach to achieve the same.