Multi task learning in tensorflow with different learning rates

user_4523769 0
Viewed 580

I want to implement a multi task learning framework in tensorflow. I'm currently implementing something like this (pseudo code)

define_reg_cost()
define_clas_cost()

reg_optimizer.(reg_learning_rate).min(reg_cost)
clas_optimizer.(clas_learning_rate).min(clas_cost)

for num_iterations:
    reg_optimizer.run()
    clas_optimizer.run()

This dosen't seem the right approach to me. I have two loss functions and I want to learn the weights to minimize both losses with different learning rates in the same optimizer. I basically want the gradient optimizer to do updates like this:

weight_updated = weight - learning_rate_1 * d(loss1)/dw - learning_rate_2 * d(loss2)/dw

Any tips on how to do this (possibly using optimizer.apply_gradients())?

tensorflow deep-learning python-3.x
1 Answers

Late but maybe helpfull for others: The solution you have shown would be exactly the right one.

If you take a look at: this post and: this stack overflow answer

If you were using something like GradientDescentOptimizer, these would be very similar operations. That's because taking gradients is a linear operation, and the gradient of a sum is the same as the sum of the gradients.

But, ADAM does something special: regardless of the scale of your loss, it scales the gradients so that they're always on the order of your learning rate. If you multiplied your loss by 1000, it wouldn't affect ADAM, because the change it would be normalized away.

using two distinct optimisers has a lot of advantages.

You also could split the update in gradient calculation and update and apply the lerningrate directly to the diffrent parts of gradients. see here

user_82253070
Related