Multilabel/ Multitask/ Multiclass Regression in machine learning

user_1340606 0
Viewed 1044

My challenge is to train a neural network to recognize certain actions and events for different classes of task or how you want to call it given the input. I see that most of the input/output when training neural networks is either 0 or 1 or [0,1]. But in my scenario I want my input to be in the form of integers which are arbitrarily big and the same form is expected for the output.

Let me give you an example: Input 

X = [ 23, 4, 0, 1233423, 1, 0, 0] ->
Y = [ 2, 1, 1]

Now each element in X[i] represent different properties of the same entity. Let's say it want to describe a human being:

23 -> maps to a place he/she was born
4 -> maps to a school they graduated

etc.

Each entry in Y[i], on the other hand, means what is more likely the human to do in 3 different categories ( as len(Y) is 3 in this case ):

Y[0] = 2 -> maps to eating icecream ( from a variety of other choices )
Y[1] = 1 -> maps to a time of day moment ( morning, noon, afternoon, evening, etc...)
Y[2] = 1 -> maps to a day of the week for example

Now of course if the task was just a multi label problem I would apply a sigmoid on the output layer and do a binary_crossentropy as the loss function but that of course does not work. Here because my output is obviously not between [0,1]. Also I am not really sure what loss function to apply since I want all classes/subclasses in Y to be correctly predicted. What I am basically saying is that each Y[i] is itself is a class of its own. It would be more accurate if my output was in the shape of (3, labels_per_class) and the loss function would calculate a loss for each of the 3 different classes trying to optimize the result in such a way that each of the 3 classes would have the correct labels. I am not sure if that is possible or how at least.

I am really still in the beginnings with my neural network knowledge and learning so clearly I am struggling with this problem.

But really to put it more simply I have a better idea how to describe it. It is more or less like an auto-encoder but the inputs and outputs are integers. The difference is that in my case the output has a different size from the input where in the auto-encoder they are the same.

My solution was to apply a relu at the output layer, ( and of course relu-like activations on all other layers as well ) and binary_crossentropy as the loss functions but the accuracy of the network is very low, around 15%.

keras tensorflow
1 Answers

For a standard classification you would probably do a dense layer with a number of nodes equal to the number of classes then apply softmax. The loss would be tf.losses.softmax_cross_entropy. You would do a sigmoid if you want to allow multiple classes, not just one.

Now you have multiple classification tasks. One way to do it is to take the last hidden layer (the one before the one where you do softmax). For each task do a dense layer with a number of nodes equals to the number of classes for that task and apply softmax. To compute the loss just add the losses together.

If the tasks are too different you may want to have more than one layer for each prediction.

You can also put some weights on the different losses if, say, eating ice-cream is a lot more important than getting the time of day right.

Only use relu if the prediction space is continous. Say time of day is continous but the choice between eating ice-cream, going to work, watching TV is not. If you use relu use a loss like L1(tf.losses.absolut_difference) or L2 (tf.losses.mean_squared_error).

user_2410130
Related