GridSearchCV with n_jobs=-1 is not working for Decision Tree/Random Forest classification

user_7366596 0
Viewed 2329

I am trying to use the GridSearchCV to evaluate different models with different parameter sets. Logistic Regression and k-NN do not cause a problem but Decision Tree, Random Forest and some of the other types of classifiers do not work when n_jobs=-1.

for classifier, paramSet, classifierName in zip(list_classifiers, list_paramSets, list_clfNames):
    gs = GridSearchCV(
                      estimator = classifier,
                      param_grid = paramSet,
                      cv = 10,      
                      n_jobs = -1           
                    )
    gs.fit(X_train, y_train)
    plot_learning_curve(gs, "Learning Curve", X_train, y_train, n_jobs=-1)

I am working on Google Colab and either of the solution proposals below did not solve my problem.

from sklearn.externals.joblib import parallel_backend
clf = GridSearchCV(...)
with parallel_backend('threading',n_jobs = -1):
    clf.fit(x_train, y_train)

if __name__ == "__main__":
    import multiprocessing as mp; mp.set_start_method('forkserver', force=True) // 'spawn' has also failed
    /// Gridsearch and fit here ///

Here is my source code : https://github.com/bahadirbasaran/pulsarDetection/blob/master/main.ipynb

The error log:

error message

Any help will be appreciated!

grid-search data-science scikit-learn multithreading python
1 Answers

you shouldn't need to use any additional calls to create threads. Your first code snippet should work. If you call:

n_cpus = multiprocessing.cpu_count()

what does it return? If you use that and then pass n_jobs=n_cpus or n_jobs=n_cpus-1 (if you don't want your computer pegged), see if that works. From the sklearn logistic regression documentation:

n_jobs int, default=None Number of CPU cores used when parallelizing over classes if multi_class=’ovr’”. This parameter is ignored when the solver is set to ‘liblinear’ regardless of whether ‘multi_class’ is specified or not.

So it may be that the models that are working are actually not using more than 1 CPU.

Hope some of these ideas help.

user_42403510
Related