How to properly use from_logits in keras loss function for binary classification?

user_11816060 0
Viewed 828

Here is a sample neural net using tensorflow,

x = tf.keras.layers.Input((None,))
x = tf.keras.layers.Dense(100)(x)
x1 = tf.keras.layers.Dense(1)(x)

model = tf.keras.Model(inputs=x, outputs=x1)

model.compile("adam", loss=tf.keras.losses.BinaryCrossentropy(from_logits=True))

model.fit(x, y)

Am I using from_logits correctly in binary loss? or should I change my output layer to,

x1 = tf.keras.layers.Dense(1, activation="sigmoid")

Can anyone clear this confusion?

I have tried the below model without from_logits in loss function, and I was getting good results, but if I use from_logits then not getting good results.

x = tf.keras.layers.Input((None,))
x = tf.keras.layers.Dense(100)(x)
x1 = tf.keras.layers.Dense(1)(x)

model = tf.keras.Model(inputs=x, outputs=x1)

model.compile("adam", loss=tf.keras.losses.BinaryCrossentropy())

model.fit(x, y)
tensorflow2.0 keras tensorflow python-3.x python
2 Answers

I've been making a little research because I've been having some similar issues when trying to replicate Tensorflow and Pytorch architectures in Tensorflow 2.0 using Keras.

In this post you can have a brief look at why the from_logits parameter exists. Briefly speaking, it seems the raw Tensorflow way to train a binary classification model was to get the output from an unbounded layer (e.g. with a linear activation function) and calculate the binary crossentropy using the "logits". In the Neural Networks context, "logits" is the output of the last unbounded layer. However, this is not the correct mathematical definition of "logits".

I've seen some posts that say that Sigmoid activation function + Binary Crossentropy in Keras for training is unstable. Nevertheless, there are some other posts that try to show that's not the case. For instance, in this post there seem to be several good reasons why it is not unstable.

In my opinion, the best approach in Keras or Tensorflow 2.0 is to use BinaryCrossentropy without logits and the sigmoid activation function in the last layer. This is more straightforward and it should work properly.

user_132483000

Yes you are using the Dense layer incorrectly.

tf.keras.layers.Dense(1, activation="sigmoid") should be used for binary classification otherwise it is linear.

Also, it might be better to choose an activation function here (x = tf.keras.layers.Dense(100)(x) ) as well, i.e. activation = 'relu' .

I suggest keeping it as default for now.

model.compile("adam", loss=tf.keras.losses.BinaryCrossentropy())

I would suggest you try optimizer= 'rmsprop' instead of 'adam'.

user_150587650
Related