indices[201] = [0,8] is out of order. Many sparse ops require sorted indices.Use `tf.sparse.reorder` to create a correctly ordered copy

user_13598116 0
Viewed 9532

Im doing a neural network encoding every variable and when im going to fit the model, an error raises.

indices[201] = [0,8] is out of order. Many sparse ops require sorted indices.
    Use `tf.sparse.reorder` to create a correctly ordered copy.

 [Op:SerializeManySparse]

I dunno how to solve it. I can print some code here and if u want more i can still printing it

def process_atributes(df, train, test):

    continuas = ['Trip_Duration']
    cs = MinMaxScaler()
    trainCont = cs.fit_transform(train[continuas])
    testCont = cs.transform(test[continuas])

    discretas = ['Start_Station_Name', 'End_Station_Name', 'User_Type', 'Genero', 'Hora_inicio']
    ohe = OneHotEncoder()
    ohe.fit(train[discretas])

    trainDisc = ohe.transform(train[discretas])
    testDisc = ohe.transform(test[discretas])

    trainX = sc.sparse.hstack((trainDisc, trainCont))
    testX = sc.sparse.hstack((testDisc, testCont))
    return (trainX, testX)

def prepare_targets(df, train, test):

    labeled_col = ['RangoEdad']

    le = LabelEncoder()
    le.fit(train[labeled_col].values.ravel())
    trainY = le.transform(train[labeled_col])
    testY = le.transform(test[labeled_col])
    return trainY, testY

X_train_enc, X_test_enc = process_atributes(dataFrameDepurado2, train, test)
Y_train_enc, Y_test_enc = prepare_targets(dataSetPrueba, train, test)

model = Sequential()
model.add(Dense(10, input_dim = X_train_enc.shape[1], activation = 'tanh', kernel_initializer = 'he_normal'))
model.add(Dense(4, activation = 'sigmoid'))

model.compile(loss = 'sparse_categorical_crossentropy', optimizer = SGD(lr = 0.01), metrics = ['accuracy'])

model.fit(X_train_enc, Y_train_enc, validation_data = (X_test_enc, Y_test_enc), epochs = 20, batch_size = 64, shuffle = True)

This is my DataSet

this is my dataSet

Thank you in advance.

keras tensorflow sparse-matrix neural-network python
1 Answers

Mentioning the solution here (Answer Section) even though it is present in the Comments Section, for the benefit of the Community.

The documentation for SparseTensor states

By convention, indices should be sorted in row-major order (or equivalently 
lexicographic order on the tuples indices[i]). This is not enforced when
SparseTensor objects are constructed, but most ops assume correct ordering. If 
the ordering of sparse tensor st is wrong, a fixed version can be obtained by
calling [tf.sparse.reorder(st)][2].

So, using either tf.sparse.reorder or scipy.sort_indices on the matrices, X_train_enc, X_test_enc, Y_train_enc, Y_test_enc, before the line of code,

model.fit(X_train_enc, Y_train_enc, validation_data = (X_test_enc, 
Y_test_enc), epochs = 20, batch_size = 64, shuffle = True)

will resolve the issue.

For more information, please refer documentation of Sparse Tensor and tf.sparse.reorder.

Hope this helps. Happy Learning!

user_134652580
Related