Performing Regression for all DataFrames Inside a List

user_17252280 0
Viewed 35

I have a list called "Data" contained with 81 DataFrames (df1,df2...df81) with each DataFrame having the same shape and label. Let's say the independent variables (X) are 'a','b','c', and the dependent variable (Y) is 'y'. Can I perform a multivariate regression on each DataFrame inside list "Data" simultaneously instead of doing it one by one? and also storing each regression accuracy (r2_score) into accuracy_list?

e.g I do regression with codes below

accuracy_list =[]
#First dataframe (df1)
X = Data['df1'][['a','b','c']]
Y = Data['df1']['y']
from sklearn.model_selection import train_test_split
X_train,X_test,Y_train,Y_test = train_test_split(X,Y,train_size=0.9,random_state=42)
from sklearn.linear_model import LinearRegression
rgs = LinearRegression()
rgs.fit(X_train,Y_train)
from sklearn.metrics import r2_score
y_pred = rgs.predict(X_test)
r2_score(Y_test,y_pred) # append it to accuracy_list

#second dataframe (df2)
X = Data['df2'][['a','b','c']]
Y = Data['df2']['y']
X_train,X_test,Y_train,Y_test = train_test_split(X,Y,train_size=0.9,random_state=42)
rgs = LinearRegression()
rgs.fit(X_train,Y_train)
y_pred = rgs.predict(X_test)
r2_score(Y_test,y_pred) # append it to accuracy_list
# and so on
pandas regression dataframe list python
1 Answers

As I understand from your question, the important part is that you can process them in parallel to speed up the computation. Therefore, you could try using multiprocessing, which spins up various processes to execute your code. One very convenient way, that is also used under the hood in sci-kit learn would be to use joblib parallel here.

In code, that would roughly read as

from joblib import Parallel, delayed
def compute_r2_score(model, X, y) -> float:
    y_pred = rgs.predict(X)
    return r2_score(y, y_pred)

n_jobs = 2 # For having 2 processes. That should be at max n_cpus - 1
# verbose=10 gives you some output on the iterations
accuracy_list = Parallel(n_jobs=n_jobs, verbose=10)(delayed(compute_r2_score)(rgs, df[['a','b','c']], df['y']) for df in data.values())

Note that multiprocessing doesn't come for free and introduces additional communication and processing overhead. Apart from that, everything that runs in an individual process must be pickable, just in case you run into that issue.

As a side note, multithreading wouldn't speed up anything here due to the Global Interpreter Lock and this task being certainly CPU bound.

user_140787580
Related