Getting error AttributeError: 'bool' object has no attribute 'transpose' when attempting to fit machine learning model

user_13519959 0
Viewed 2746

I am trying to create a machine learning model to predict who would survive on the Titanic. Everytime I try to fit my model, I get this error :

Traceback (most recent call last):
  File "c:\Users\seand\.vscode\extensions\ms-python.python-2020.6.89148\pythonFiles\ptvsd_launcher.py", line 48, in <module>
    main(ptvsdArgs)
  File "c:\Users\seand\.vscode\extensions\ms-python.python-2020.6.89148\pythonFiles\lib\python\old_ptvsd\ptvsd\__main__.py", line 432, in main
    run()
  File "c:\Users\seand\.vscode\extensions\ms-python.python-2020.6.89148\pythonFiles\lib\python\old_ptvsd\ptvsd\__main__.py", line 316, in run_file
    runpy.run_path(target, run_name='__main__')
    return _run_module_code(code, init_globals, run_name,
  File "D:\Python\lib\runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "D:\Python\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "d:\Kaggle\Titanic\titanic4.py", line 100, in <module>
    cat_cols2 = pd.DataFrame(OneHot1.fit_transform(new_df[cat_columns]))
  File "D:\Python\lib\site-packages\pandas\core\frame.py", line 2806, in __getitem__
    indexer = self.loc._get_listlike_indexer(key, axis=1, raise_missing=True)[1]
  File "D:\Python\lib\site-packages\pandas\core\indexing.py", line 1552, in _get_listlike_indexer
    self._validate_read_indexer(
  File "D:\Python\lib\site-packages\pandas\core\indexing.py", line 1640, in _validate_read_indexer
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index(['Name', 'Sex', 'Ticket', 'Cabin', 'Embarked'], dtype='object')] are in the [columns]"
PS D:\Kaggle\Titanic>  cd 'd:\Kaggle\Titanic'; ${env:PYTHONIOENCODING}='UTF-8'; ${env:PYTHONUNBUFFERED}='1'; & 'D:\Python\python.exe' 'c:\Users\seand\.vscode\extensions\ms-python.python-2020.6.89148\pythonFiles\ptvsd_launcher.py' '--default' '--client' '--host' 'localhost' '--port' '60778' 'd:\Kaggle\Titanic\titanic4.py'
Traceback (most recent call last):
  File "c:\Users\seand\.vscode\extensions\ms-python.python-2020.6.89148\pythonFiles\ptvsd_launcher.py", line 48, in <module>
    main(ptvsdArgs)
  File "c:\Users\seand\.vscode\extensions\ms-python.python-2020.6.89148\pythonFiles\lib\python\old_ptvsd\ptvsd\__main__.py", line 432, in main
    run()
  File "c:\Users\seand\.vscode\extensions\ms-python.python-2020.6.89148\pythonFiles\lib\python\old_ptvsd\ptvsd\__main__.py", line 316, in run_file
    runpy.run_path(target, run_name='__main__')
  File "D:\Python\lib\runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "D:\Python\lib\runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "D:\Python\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "d:\Kaggle\Titanic\titanic4.py", line 143, in <module>
    my_pipeline.fit(new_df,y)
  File "D:\Python\lib\site-packages\sklearn\pipeline.py", line 330, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "D:\Python\lib\site-packages\sklearn\pipeline.py", line 292, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "D:\Python\lib\site-packages\joblib\memory.py", line 352, in __call__
    return self.func(*args, **kwargs)
  File "D:\Python\lib\site-packages\sklearn\pipeline.py", line 740, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "D:\Python\lib\site-packages\sklearn\compose\_column_transformer.py", line 531, in fit_transform
    result = self._fit_transform(X, y, _fit_transform_one)
  File "D:\Python\lib\site-packages\sklearn\compose\_column_transformer.py", line 458, in _fit_transform
    return Parallel(n_jobs=self.n_jobs)(
  File "D:\Python\lib\site-packages\joblib\parallel.py", line 1032, in __call__
    while self.dispatch_one_batch(iterator):
  File "D:\Python\lib\site-packages\joblib\parallel.py", line 847, in dispatch_one_batch
    self._dispatch(tasks)
  File "D:\Python\lib\site-packages\joblib\parallel.py", line 765, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "D:\Python\lib\site-packages\joblib\_parallel_backends.py", line 206, in apply_async
    result = ImmediateResult(func)
  File "D:\Python\lib\site-packages\joblib\_parallel_backends.py", line 570, in __init__
    self.results = batch()
  File "D:\Python\lib\site-packages\joblib\parallel.py", line 252, in __call__
    return [func(*args, **kwargs)
  File "D:\Python\lib\site-packages\joblib\parallel.py", line 252, in <listcomp>
    return [func(*args, **kwargs)
    res = transformer.fit_transform(X, y, **fit_params)
  File "D:\Python\lib\site-packages\sklearn\pipeline.py", line 367, in fit_transform
    Xt = self._fit(X, y, **fit_params_steps)
  File "D:\Python\lib\site-packages\sklearn\pipeline.py", line 292, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "D:\Python\lib\site-packages\joblib\memory.py", line 352, in __call__
    return self.func(*args, **kwargs)
  File "D:\Python\lib\site-packages\sklearn\pipeline.py", line 740, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "D:\Python\lib\site-packages\sklearn\base.py", line 693, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "D:\Python\lib\site-packages\sklearn\impute\_base.py", line 459, in transform
    coordinates = np.where(mask.transpose())[::-1]
AttributeError: 'bool' object has no attribute 'transpose'
PS D:\Kaggle\Titanic>  cd 'd:\Kaggle\Titanic'; ${env:PYTHONIOENCODING}='UTF-8'; ${env:PYTHONUNBUFFERED}='1'; & 'D:\Python\python.exe' 'c:\Users\seand\.vscode\extensions\ms-python.python-2020.6.89148\pythonFiles\ptvsd_launcher.py' '--default' '--client' '--host' 'localhost' '--port' '60800' 'd:\Kaggle\Titanic\titanic4.py' 
Traceback (most recent call last):
  File "c:\Users\seand\.vscode\extensions\ms-python.python-2020.6.89148\pythonFiles\ptvsd_launcher.py", line 48, in <module>
    main(ptvsdArgs)
  File "c:\Users\seand\.vscode\extensions\ms-python.python-2020.6.89148\pythonFiles\lib\python\old_ptvsd\ptvsd\__main__.py", line 432, in main
    run()
  File "c:\Users\seand\.vscode\extensions\ms-python.python-2020.6.89148\pythonFiles\lib\python\old_ptvsd\ptvsd\__main__.py", line 316, in run_file
    runpy.run_path(target, run_name='__main__')
  File "D:\Python\lib\runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "D:\Python\lib\runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "D:\Python\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "d:\Kaggle\Titanic\titanic4.py", line 122, in <module>
    my_pipeline.fit(new_df,y)
  File "D:\Python\lib\site-packages\sklearn\pipeline.py", line 330, in fit
    Xt = self._fit(X, y, **fit_params_steps)
  File "D:\Python\lib\site-packages\sklearn\pipeline.py", line 292, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "D:\Python\lib\site-packages\joblib\memory.py", line 352, in __call__
    return self.func(*args, **kwargs)
  File "D:\Python\lib\site-packages\sklearn\pipeline.py", line 740, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "D:\Python\lib\site-packages\sklearn\compose\_column_transformer.py", line 531, in fit_transform
    result = self._fit_transform(X, y, _fit_transform_one)
  File "D:\Python\lib\site-packages\sklearn\compose\_column_transformer.py", line 458, in _fit_transform
    return Parallel(n_jobs=self.n_jobs)(
  File "D:\Python\lib\site-packages\joblib\parallel.py", line 1032, in __call__
    while self.dispatch_one_batch(iterator):
  File "D:\Python\lib\site-packages\joblib\parallel.py", line 847, in dispatch_one_batch
    self._dispatch(tasks)
  File "D:\Python\lib\site-packages\joblib\parallel.py", line 765, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "D:\Python\lib\site-packages\joblib\_parallel_backends.py", line 206, in apply_async
    result = ImmediateResult(func)
  File "D:\Python\lib\site-packages\joblib\_parallel_backends.py", line 570, in __init__
    self.results = batch()
  File "D:\Python\lib\site-packages\joblib\parallel.py", line 252, in __call__
    return [func(*args, **kwargs)
  File "D:\Python\lib\site-packages\joblib\parallel.py", line 252, in <listcomp>
    return [func(*args, **kwargs)
  File "D:\Python\lib\site-packages\sklearn\pipeline.py", line 740, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "D:\Python\lib\site-packages\sklearn\pipeline.py", line 367, in fit_transform
    Xt = self._fit(X, y, **fit_params_steps)
  File "D:\Python\lib\site-packages\sklearn\pipeline.py", line 292, in _fit
    X, fitted_transformer = fit_transform_one_cached(
  File "D:\Python\lib\site-packages\joblib\memory.py", line 352, in __call__
    return self.func(*args, **kwargs)
  File "D:\Python\lib\site-packages\sklearn\pipeline.py", line 740, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)
  File "D:\Python\lib\site-packages\sklearn\base.py", line 693, in fit_transform
    return self.fit(X, y, **fit_params).transform(X)
  File "D:\Python\lib\site-packages\sklearn\impute\_base.py", line 459, in transform
    coordinates = np.where(mask.transpose())[::-1]
AttributeError: 'bool' object has no attribute 'transpose'

The code I am running is the following :


from xgboost import XGBClassifier
from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.feature_selection import SelectFromModel
from itertools import combinations
import pandas as pd 
import numpy as np

#read in data
training_data = pd.read_csv('train.csv')
testing_data = pd.read_csv('test.csv')




#seperate X and Y
X_train_full = training_data.copy()
y = X_train_full.Survived
X_train_full.drop(['Survived'], axis=1, inplace=True)

y_test = testing_data

#get all str columns
cat_columns1 = [cname for cname in X_train_full.columns if
                    X_train_full[cname].dtype == "object"]

interactions = pd.DataFrame(index= X_train_full)

#create new features
for combination in combinations(cat_columns1,2):
    imputer = SimpleImputer(strategy='constant')

    new_col_name = '_'.join(combination)
    col1 = X_train_full[combination[0]]
    col2 = X_train_full[combination[1]]
    col1 = np.array(col1).reshape(-1,1)
    col2 = np.array(col2).reshape(-1,1)
    col1 = imputer.fit_transform(col1)
    col2 = imputer.fit_transform(col2)


    new_vals = col1 + '_' + col2
    OneHot = OneHotEncoder()




    interactions[new_col_name] = OneHot.fit_transform(new_vals)
 

interactions = interactions.reset_index(drop = True)


#create new dataframe with new features included
new_df = X_train_full.join(interactions)
 

#do the same for the test file
interactions2 = pd.DataFrame(index= y_test)
for combination in combinations(cat_columns1,2):
    imputer = SimpleImputer(strategy='constant')

    new_col_name = '_'.join(combination)
    col1 = y_test[combination[0]]
    col2 = y_test[combination[1]]
    col1 = np.array(col1).reshape(-1,1)
    col2 = np.array(col2).reshape(-1,1)
    col1 = imputer.fit_transform(col1)
    col2 = imputer.fit_transform(col2)


    new_vals = col1 + '_' + col2

    OneHot = OneHotEncoder()




    interactions2[new_col_name] = OneHot.fit_transform(new_vals)


    interactions2[new_col_name] = new_vals
 

interactions2 = interactions2.reset_index(drop = True)
y_test = y_test.join(interactions2)


#get names of cat columns (with new features added)
cat_columns = [cname for cname in new_df.columns if
                    new_df[cname].dtype == "object"]

# Select numerical columns
num_columns = [cname for cname in new_df.columns if 
                new_df[cname].dtype in ['int64', 'float64']]



#set up pipeline
numerical_transformer = SimpleImputer(strategy = 'constant')


categorical_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='constant')),
    ('onehot', OneHotEncoder(handle_unknown='ignore'))
])


preprocessor = ColumnTransformer(
    transformers=[
        ('num', numerical_transformer, num_columns),
        ('cat', categorical_transformer, cat_columns)
    ])
model = XGBClassifier()

my_pipeline = Pipeline(steps=[('preprocessor', preprocessor),
                              ('model', model)
                             ])
#fit model
my_pipeline.fit(new_df,y)

The csv files I am reading are available from Kaggle at this link :

https://www.kaggle.com/c/titanic/data

I cannot figure out what is causing this problem. Any help would be much appreciated.

scikit-learn pandas machine-learning model python
1 Answers

This probably happens because your data contains pd.NA values. pd.NA was introduced in pandas 1.0.0, but is still marked as experimental.

SimpleImputer will ultimately run data == np.nan, which would usually return a numpy array. In stead, it is returning a single boolean scalar when data contains pd.NA values.

An example:

import pandas as pd
import numpy as np

test_pd_na = pd.DataFrame({"A": [1, 2, 3, pd.NA]})
test_np_nan = pd.DataFrame({"A": [1, 2, 3, np.nan]})

test_np_nan.to_numpy() == np.nan:
> array([[False],
       [False],
       [False],
       [False]])

test_pd_na.to_numpy() == np.nan

> False

The solution would be to convert all pd.NA values to np.nan before running SimpleImputer. You can use .replace({pd.NA: np.nan})on your data frames for this purpose. The downside is obviously that you loose the benefits pd.NA brings, such as integer columns with missing data, in stead of those columns being converted to float columns.

user_140998940
Related