ValueError in Titanic pipeline

user_13735281 0
Viewed 101

I'm working on my first pipeline but I can't get it to work on the Titanic dataset. Can someone please explain what I'm doing wrong and how I can fix it?

I dropped some features from the dataframe and used get dummies to transform categorical features.

titanic_dummies = titanic.copy()
titanic_dummies = titanic_dummies.drop([ 'Name', 'Ticket','Cabin', "Fare"], axis=1)
titanic_dummies = pd.get_dummies(titanic_dummies, drop_first=True)

Then I tried to run this pipeline

X=titanic_dummies.drop(['Survived'], axis=1)
y=titanic_dummies['Survived']


 ****#setup the pipeline steps****
steps = [('scaler', StandardScaler()),
         ('imputation', SimpleImputer(missing_values='NaN', strategy='most_frequent')),
         ('logreg', LogisticRegression())]
          
*# Create the pipeline: pipeline*
pipeline = Pipeline(steps)

#Define hyperparameters and range of Grid Search
parameters = {"logreg__C": np.logspace(-5, 8, 15),
              "logreg__penalty": ['l1', 'l2']}

*# Create train and test sets*
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

*# run cross validation*
cv = GridSearchCV(pipeline, param_grid = parameters, cv=3)

*# Fit the pipeline to the training set:* 
cv.fit(X_train, y_train)

*# Predict the labels of the test set*
y_pred = cv.predict(X_test)

*# Compute and print metrics*
print("Accuracy: {}".format(cv.score(X_test, y_test)))
print(classification_report(y_test, y_pred))
print("Tuned Model Parameters: {}".format(cv.best_params_))

This is the error that I get

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

From the attached picture you can probably tell that the size of my values are not the issue. Perhaps something goes wrong with my imputation?

Titanic Data

I really love to hear your thoughts on how I can fix this.

valueerror pipeline python
3 Answers

You may change

missing_values = "NaN"

to

missing_values = np.nan

this might work.

user_131052810

The NaN's in the numpy arrray are represented as np.NaN not as string type "NaN"

Fix

('imputation', SimpleImputer(missing_values=np.NaN, strategy='most_frequent'))
user_4239260

This can happen if you have any unwanted records. What I mean by this is that even though the dtypes say float, the Nans are a problem for your program. You have a couple of options here:

  1. Delete such records
  2. Fill them with the mean/median value (I wouldn't recommend it for titanic)
  3. Replace them with a default value
user_71224380
Related