Sparse categorical cross entropy not working as expected

user_8520011 0
Viewed 506

I am working on MIMIC 3 data set to classify patients. I have 7000 classes and the range of values of class is upto 50000, labels are integers. So I used sparse categorical entropy loss so that I dont have to worry about high range of labels. But I my model gives a nan loss on the labels greater than 7000. 

n_timesteps, n_features, n_outputs = 50, 59, 7000
batch_size=32
model = Sequential()
model.add(CuDNNLSTM(32, input_shape=(n_timesteps,n_features)))
model.add(Dropout(0.3))
model.add(Dense(256, activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(n_outputs, activation='softmax'))
model.compile(loss=keras.losses.sparse_categorical_crossentropy, optimizer=keras.optimizers.Adam(), metrics=['sparse_categorical_accuracy'])

I am normalizing the data by 

def normalize(df):

  header=df.columns
  subid_col=df['SUBJECT_ID']
  label_col=df['label']
  df = df.replace(0, np.NaN)
  df=pd.DataFrame(df).fillna(method="ffill")
  df=pd.DataFrame(df).fillna(df.mean())
  df=pd.DataFrame(df).fillna(-99)
  x = df.values
  header=df.columns

  min_max_scaler = preprocessing.MinMaxScaler(feature_range = (1,2))
  x_scaled = min_max_scaler.fit_transform(x)
  df = pd.DataFrame(x_scaled)
  df=pd.DataFrame(df).fillna(-1)

  df.columns=header

  df['SUBJECT_ID']=subid_col
  df['label']=label_col
  df.drop(df.columns[0], axis = 1, inplace = True)

  return df

I am doing a time series classification.

keras classification time-series machine-learning python
1 Answers

Since your final layer is of length n_outputs=7000, sparse_categorical_crossentropy expects labels in index notation in the range [0, 7000]. You thus need to map your labels before training i.e:

from sklearn import preprocessing

le = preprocessing.LabelEncoder()
le.fit(label_col)

df['label'] = le.transform(label_col)

To retrieve the 'true' labels from the encoding you can use le.inverse_transform(label_col).

Alternatively you could just change your n_outputs = 50000, but this would arguably be inefficient if you are only training on / expecting to see a subset of 7000 of these, since you are unnecessarily adding parameters which would ideally be learned as 0, since you have no data with these labels.

user_90676150
Related