The question could be quite trivial, but I was wondering if anyone could help me parallel process this loop. There is a reference to "n" (row number) and a CSV which I print the answers too... I am unsure how to (if I need to) handle this "n" value between the parallelism?
I have googled a number of topics and checked stackoverflow for examples - I was hoping to use the multiprocessing library, but I'm happy to stand corrected if something else is better. My Python skill is relatively basic and I have never looked into parallel processing before.
import multiprocessing as mp
import numpy as np
import pandas as pd
import seaborn as sns
from fitter import Fitter#, get_common_distributions, get_distributions
from fitter import get_common_distributions
from fitter import get_distributions
import os
import IPython
dataset = pd.read_csv("econ.csv")
df = pd.DataFrame({'Name': pd.Series([], dtype='str'),
'Output': pd.Series([], dtype='str')})
n=0
for col in dataset.columns:
spac = dataset[col].values
f = Fitter(spac, distributions=get_distributions())
f.fit()
print(col,n)
df.loc[n]=[col]+[f.get_best(method='sumsquare_error')]
df.to_csv('dist.csv', index=False)
n=n+1
print(df)
IPython.Application.instance().kernal.do_shutdown(True)
An example of the output file (dist.csv) is:
| Name | Output |
|---|---|
| SPAC | {'gennorm': {'beta': 1.1103780008768631, 'loc': 0.0977722086198811, 'scale': 0.8284342569644088}} |