Scikit-learn APIs are not natively designed for your use case. I'm also not aware of a single high-level function that does what you're describing. However, this is a great problem for Dask, which will allow you to parallelize this quite nicely with very few changes to your code.
Dask's delayed API allows you to smoothly parallelize for-loops like in your example. I highly recommend checking out the introductory tutorial.
Original:
import pandas as pd
import numpy as np
from sklearn.linear_model import LinearRegression
np.random.seed(12)
df = pd.DataFrame({
'y': np.random.randn(20),
'x1': np.random.randn(20),
'x2': np.random.randn(20),
'x3': np.random.randn(20)})
from sklearn.linear_model import LinearRegression
for i in range(3):
t=LinearRegression().fit(df[['x'+str(i+1)]],df[['y']])
print(t.coef_, t.intercept_)
[[0.12973704]] [0.11022991]
[[0.09058823]] [0.02903383]
[[0.12020571]] [0.02941156]
Dask Version (using delayed):
import pandas as pd
import numpy as np
from sklearn.linear_model import LinearRegression
from dask import delayed
from dask.distributed import Client
import dask
client = Client() # set up a local Dask cluster
def fit_ols(df, feature_col, target_col="y"):
clf = LinearRegression()
clf.fit(df[[feature_col]], df[target_col])
return clf.coef_[0], clf.intercept_
results = []
for i in range(3):
xcol = f"x{i+1}"
res = delayed(fit_ols)(df, xcol)
results.append(res)
dask.compute(results)
([(0.1297370368274274, 0.11022990787315554),
(0.09058823367556357, 0.029033826351961462),
(0.12020570872010947, 0.02941156049324152)],)
The key change is wrapping your logic into a function. With that, we can rely on delayed to add tasks to a task graph and then call compute on the graph to execute the tasks in parallel.
Editing to include an example of how you might want to do this at a larger scale, since you have millions of columns. Note the scatter operation and passing of the future.
import pandas as pd
import numpy as np
from sklearn.linear_model import LinearRegression
from dask import delayed
from dask.distributed import Client, LocalCluster
import dask
# set up a local cluster
# my laptop has limited CPU cores, so only using two to parallelize
client = Client(n_workers=2, threads_per_worker=1)
# Setup data
np.random.seed(12)
nrows = 10000
ncols = 1000
df = pd.DataFrame({f"x{i}": np.random.randn(nrows) for i in range(ncols)})
df['y'] = np.random.randn(nrows)
# scatter the data to the workers beforehand
data_future = client.scatter(df, broadcast=True)
# Many calls to linear regression, parallelized with Dask
def fit_ols(df, feature_col, target_col="y"):
clf = LinearRegression()
clf.fit(df[[feature_col]], df[target_col])
return clf.coef_[0], clf.intercept_
results = []
for i in range(df.shape[1]-1):
xcol = f"x{i}"
# note how i'm passing the scattered data future
res = delayed(fit_ols)(data_future, xcol)
results.append(res)
res = dask.compute(results)
res[0][:5]
[(0.01898470494963711, -0.0013691188314244067),
(-0.018210714678412274, -0.0015618953728564272),
(-0.013344320263479937, -0.001588615271127849),
(-0.006330810820098386, -0.0016927025190818918),
(0.007960791021720603, -0.0016594945951821688)]
I've attached a screenshot of the dashboard showing the computation in progress:
