Before going into the solution I would first comment on the proposed solution of the questions. The first solution would work better compared to the second. This is because It is very hard to interpret the (probability )values of the neural network output. Closeness of the values might be caused by similarity of the classes involving(in this case a dog might look like a cat). Sometimes you may end up getting unseen classes being assigned to one of the classes with high probability.
Most of supervised classification machine learning algorithms are designed to map an input to one of some fixed number of classes. This type of classification is called closed world classification.
E.g.
- MNIST - handwritten digit classification
- Cat - Dog classification
When classification involves some unlabeled/unknown classes, the approach is called Open-world classification. There are various papers published[1, 2, 3].
I will explain my solution using the solution proposed by 3.
There are two options to apply the Open world classification(Here on I will refer to OWC) to the problem in question.
- Classifying all new classes as single class
- Classifying all new classes as single class, then further grouping similar samples into single class and different samples into different classes.
1. Classifying all new classes as single class
Although there could be many types of model that could fit to this type of classification(One of could be the first solution proposed by the question.) I would discusses model of 3. Here the network first decides to classify or to reject the input. Ideally if the sample is from seen classes then the network will classify into one of seen classes. Other wise the network rejects. The authors of 3 called this network Open classification network(OCN). Keras implementation of OCN could be(I've simplified the network to just focus on output of the model.
inputs = keras.layers.Input(shape=(28, 28,1))
x = keras.layers.Conv2D(64, 3, activation="relu")(inputs)
x = keras.layers.Flatten()(x)
embedding = keras.layers.Dense(256, activation="linear", name="embedding_layer")(x)
reject_output = keras.layers.Dense(1, activaton="sigmoid", name="reject_layer")(embedding)
classification_output = keras.layers.Dense(num_of_classes, activaton="softmax", name="reject_layer")(embedding)
ocn_model = keras.models.Model(inputs=inputs, outputs=[reject_output, classification_output)
The model is trained in a way that jointly optimizes both reject_output and classification_output losses.
2. Classifying all new classes as single class, then further grouping similar
The authors of 3 used another network to find similarity between samples. They called the network Pairwise Classification Network(PCN). PCN classifies whether two inputs are from the same classes or different classes. We can use the embedding of the first solution and use pairwise similarity metrics to create PCN network. In PCN the weights are shared for both inputs. This could be implemented using keras
embedding_model = keras.layers.Sequential([
keras.layers.Conv2D(64, 3, activation="relu", input_shape=(28, 28,1))
keras.layers.Flatten(),
embedding = keras.layers.Dense(256, activation="linear", name="embedding_layer")
])
input1 = keras.layers.Input(shape=(28, 28, 1))
input2 = keras.layers.Input(shape=(28, 28, 1))
embedding1 = embedding_model(input1)
embedding2 = embedding_model(input2)
merged = keras.layers.Concatenate()([embedding1, embedding2])
output = keras.layers.Dense(1, activation="sigmoid")(merged)
pcn_model = keras.models.Model(inputs=[input1, input2], outputs=output)
PCN model will be trained to reduce the distance from the same and increase the distance between different classes.
After the PCN network is trained auto-encoder is trained to learn useful representations from the unseen classes. Then Clustering algorithm is used to group(cluster) unseen classes by using PCN model as distance function.
