Concatenate array with same number of rows but different columns

user_11642229 0
Viewed 336
def preprocess(numerical , categorical):
    imputer = SimpleImputer()
    x_num = imputer.fit_transform(numerical)
    scaler = StandardScaler()
    x_num = scaler.fit_transform(x_num)
    one_hot = OneHotEncoder()
    x_cat = one_hot.fit_transform(categorical)
    print('X_num Shape : ' , x_num.shape)
    print('X_cat Shape : ' , x_cat.shape)
    
    return np.concatenate((x_num,x_cat),axis = 1)

[Output] X_num Shape :  (889, 2)
         X_cat Shape :  (889, 22)

The Error it shows at the end is ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 2 dimension(s) and the array at index 1 has 0 dimension(s)

I want the output to be of shape (889,24)

last sentence ( array at index 1 has 0 dimensions ) drives me to think that the problem is related to the weird numpy arrays of shape (n,) and (,n) but that shouldn't be a problem as dimensions are shown to not be that way but I think there's something I'm missing

I've also tried using a lot of different functions np.hstack , np.vstack , np.column_stack but they either dont give the desired output or show this error message ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 2 and the array at index 1 has size 1

numpy arrays python
1 Answers

So as hpaulj mentioned above the problem was that the type of x_cat after coming out of OneHotEncoder was <class 'scipy.sparse.csr.csr_matrix'> instead of a numpy array which cannot be concatenated with other numpy array which results in it being cast into a numpy array when concatenating and it's dimension was () using the shape command. not sure if this means it was flattened or not but when I tried to use reshape , it didnt work and said ValueError: cannot reshape array of size 1 into shape (889,22)

What I did to solve this was replace the line one_hot = OneHotEncoder() with one_hot = OneHotEncoder(sparse = False) which makes the type of output matrix a dense numpy array which can be concatenated using np.concatenate((x_num,x_cat),axis = 1 )

user_116422290
Related