Scikit-Learn PipeLine Fit and Transform Error

user_12849625 0
Viewed 855

  1. I am new to Machine Learning and practicing the codes on the book named "Hands-On Machine Learning with Scikit-Learn and TensorFlow" by Aurélien Géron. There is code regarding Pipeline which throwing an error named "AttributeError: 'longitude' is not a valid function for 'Series' object".

  2. Here is the code that I am trying to run-


class DataFrameSelector(BaseEstimator, TransformerMixin):
    def __init__(self, attribute_names):
        self.attribute_names = attribute_names
    def fit(self, X, y=None):
        return X
    def transform(self, X):
        return X[self.attribute_names].values

num_attributes = list(housing_num)
cat_attributes = ['ocean_proximity']

num_pipeline = Pipeline([ ('selector', DataFrameSelector(num_attributes)),
                         ('imputer', SimpleImputer(strategy='median')), 
                         ('Attribute_adder', CombinedAttributesAdder()), 
                         ('std_scaler', StandardScaler()) ])

cat_pipeline = Pipeline([ ('selector', DataFrameSelector(cat_attributes)),
                         ('label_binarizer', OneHotEncoder()) ])

full_pipeline = FeatureUnion( transformer_list=[ ('num_pipeline', num_pipeline), ('cat_pipeline', cat_pipeline) ] )

housing_prepared = cat_pipeline.fit_transform(housing)
housing_prepared

  1. Here is the link of the data that I am using: https://raw.githubusercontent.com/ageron/handson-ml/master/datasets/housing/housing.csv

  2. When I re-run the code without the DataFrameSelector, it works just fine. So, I looked for a possible solution and found that instead of using the FeatureUnion class ColumnTransformer class would resolve the issue which works without the need of the DataFrameSelector class. That's really great and less time consuming but didn't find any solution about what's wrong with the above code. Any help would be very much appreciated regarding this issue.

scikit-learn machine-learning python
1 Answers

In sklearn, the fit method needs to return self. That is to accommodate the sort of chaining
estimator.fit(X,y).transform(X),
which is furthermore in most cases the same as calling
estimator.fit_transform(X,y),
which is used extensively when fitting pipelines.

So, when your pipeline tries to fit_transform the DataFrameSelector, it tries to call the method transform on the dataframe X. As it turns out, dataframes do have a method transform, by which the argument callable is applied to each column series. Now, the argument of pd.DataFrame.transform can be a list (of functions to apply); in a bizarre twist, in this case that argument is a dataframe, but the code just notices that it's iterable, so it tries to loop over the columns of X and apply their names as functions. Hence 'longitude' is not a valid function.

user_104958930
Related