Why is cross_validate returning NaN scores?

Question

The data is from: https://www.kaggle.com/andieminogue/newspaper-churn

The code is shown below df_og is the loaded dataframe, some of the columns have been removed and NaN rows removed, other than that, the data hasn't been transformed:

df_og2 = df_og.drop(['Address','State','City','SubscriptionID','Zip Code'], axis = 1)
df_og2 = df_og2.dropna()
df_og2.reset_index(drop=True, inplace=True)

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import LabelEncoder, OneHotEncoder, OrdinalEncoder
from sklearn.compose import ColumnTransformer
from sklearn.base import BaseEstimator, TransformerMixin
from category_encoders import ordinal
from scipy.sparse import csr_matrix
from sklearn.preprocessing import StandardScaler
from imblearn.under_sampling import RandomUnderSampler
from sklearn.model_selection import cross_validate
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.metrics import classification_report
from imblearn.pipeline import Pipeline
from sklearn.inspection import permutation_importance
from imblearn.under_sampling import RandomUnderSampler

X = df_og2.drop(['Subscriber'], axis=1)
y = df_og2[['Subscriber']]
X, y = RandomUnderSampler(random_state=2).fit_resample(X,y)

numeric = df_og2.select_dtypes(include=['int64']).columns.tolist()
multi_label_cat = ['HH Income', 'Age range', 'weekly fee']
binary_label_cat = ['Home Ownership', 'dummy for Children']
onehot_cat = ['County','Deliveryperiod','Ethnicity','Language','Nielsen Prizm', 'Source Channel']

def getScores(model):
    
    HH = df_og2['HH Income'].unique().tolist()
    AR = df_og2['Age range'].unique().tolist()
    WF = df_og2['weekly fee'].unique().tolist()
    HH.sort()
    HH.insert(0,HH.pop())
    AR.sort()
    WF.sort()
    WF.append(WF.pop(5))
    HH_map = {i:j for i,j in zip(HH,[i for i in range(len(HH))])}
    AR_map = {i:j for i,j in zip(AR,[i for i in range(len(AR))])}
    WF_map = {i:j for i,j in zip(WF,[i for i in range(len(WF))])}
    col_map = [{'col': 'HH Income', 'mapping': HH_map}, {'col': 'Age range', 'mapping': AR_map}, 
                                                        {'col': 'weekly fee', 'mapping': WF_map}]
    encode_pipeline = ColumnTransformer([
            ('multi_label_cat', ordinal.OrdinalEncoder(mapping=col_map), multi_label_cat),
            ('binary_label_cat', OrdinalEncoder(), binary_label_cat),
            ('onehot_cat', OneHotEncoder(), onehot_cat),
    ], remainder = 'passthrough', sparse_threshold = 0)
    
    full_pipeline = Pipeline([
            ('encode',encode_pipeline),
            ('scale', StandardScaler()),
            ('model', model)
    ])

    cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=2)
    scoring = {'acc': 'accuracy',
               'f1': 'f1',
               'prec': 'precision',
               'rec': 'recall'}
    scores = cross_validate(full_pipeline, X, y, scoring=scoring , cv=cv, n_jobs=-1)
    print(scores)

from sklearn.linear_model import LogisticRegression
getScores(LogisticRegression())

Sometimes changing the number of folds and repeats to large numbers yields some values (in the same procedure) in the score with most of them being NaN. However after trying to fix it, I can't reproduce that phenomenon anymore.

user_12398468 · Answer

I solved the problem by using cross_val_score which shows error messages. There were 2 errors, the first was an issue with OneHot encoding which for some odd reason did not encode the unique values with only one row associated with them, adding the handle_unknown='ignore' parameter to the OneHotEncoder() fixed this. Another issue encounter after is invalid pos_label=1 can only be one of ['YES','NO'], I made the assumption that ordinal encoding of the y is done automatically by the cross_validate() function when that is not true. I fixed this by encoding y beforehand.

user_2174098 · Answer

To get repeatable results with RepeatedStratifiedKFold it is best to set the random_state parameter. You could just edit this line of your code:

cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=2, random_state=42)

Doing that you should be able to find settings that reduce or eliminate NaN's in the output. However, to get a more detailed understanding of why this sometimes happens, and to get better confidence that you can use your model to predict on test data outside of this initial dataset you will need to do further Exploratory Data Analysis on your dataset to understand the cause of the issue.

Why is cross_validate returning NaN scores?

2 Answers