Parallel Process Python Loop while writing to file

Viewed 44

The question could be quite trivial, but I was wondering if anyone could help me parallel process this loop. There is a reference to "n" (row number) and a CSV which I print the answers too... I am unsure how to (if I need to) handle this "n" value between the parallelism?

I have googled a number of topics and checked stackoverflow for examples - I was hoping to use the multiprocessing library, but I'm happy to stand corrected if something else is better. My Python skill is relatively basic and I have never looked into parallel processing before.

import multiprocessing as mp
import numpy as np
import pandas as pd
import seaborn as sns
from fitter import Fitter#, get_common_distributions, get_distributions
from fitter import get_common_distributions
from fitter import get_distributions
import os
import IPython

dataset = pd.read_csv("econ.csv")

df = pd.DataFrame({'Name': pd.Series([], dtype='str'),
                   'Output': pd.Series([], dtype='str')})

n=0

for col in dataset.columns:

    spac = dataset[col].values
    f = Fitter(spac, distributions=get_distributions())
    f.fit()

    print(col,n)

    df.loc[n]=[col]+[f.get_best(method='sumsquare_error')]
    df.to_csv('dist.csv', index=False)

    n=n+1
    print(df)

    IPython.Application.instance().kernal.do_shutdown(True)

An example of the output file (dist.csv) is:

Name Output
SPAC {'gennorm': {'beta': 1.1103780008768631, 'loc': 0.0977722086198811, 'scale': 0.8284342569644088}}
0 Answers
Related