ColumnTransformer : 1D data passed to a transformer that expects 2D data

user_9876913 0
Viewed 2489

It is the first time I am using ColumnTransformer and I face an issue with the dimension of the array.

Here is my dataframe called 'concat_data' :

          Date  Hour  Observation
0  01/09/2015      0           10
1  01/09/2015      1            9
2  01/09/2015      2            8
3  01/09/2015      3           10
4  01/09/2015      4           12

I am trying to apply a standard scaler only on the Observation column with the following code :

preprocessor = ColumnTransformer(
        transformers=[
            ('num', StandardScaler(), ['Observation'])],
        remainder="passthrough")

preprocessor.fit(concat_data, concat_data['Observation'])

I get this error message :

ValueError: 1D data passed to a transformer that expects 2D data. Try to specify the column selection as a list of one item instead of a scalar.

I could do it without ColumnTransformer but my objective is to understand how ColumnTransformer works. For instance, this code works :

StandardScaler().fit(concat_data[['Observation']])

Thanks all !

scikit-learn python
3 Answers

Try

preprocessor.fit(concat_data, concat_data)

or

preprocessor.fit(concat_data, concat_data[['Observation']])

Also, might be useful: example of using the ColumnTransformer

About using df['A'] and df[['A']]

In one case You still have a pd.DataFrame - using df[['A']], in the other case it is squeezed to pd.Series - df['A'].

pd.Series is considered as 1D object, because it is allowed to have one column; pd.DataFrame is a 2D object, allowed to have multiple column and treated accordingly. 

df = pd.DataFrame(data=[[1,2,3,4,5],[5,4,3,2,1]],index=[0,1],columns=['A','B','C','D','E'])

type(df[['A']])

Out[27]: pandas.core.frame.DataFrame

type(df['A'])

Out[28]: pandas.core.series.Series

The output for both cases is also slightly different.

user_91082390

You should use preprocessor.fit(concat_data) directly and it will work, the ColumnTransformer expect only one DataFrame not two. And it will automatically look for the Column Observation

user_147758840

Artur Sokolovsky, I also tried the scikit-learn's example and it also didn't work

As Vivek Kumar suggested, the issue was due to an old version of scikit-learn. Updating to 0.20.02 solved it.

user_98769130
Related