Interpolate weights in TensorFlow

user_2933468 0
Viewed 130

A have two models with the same architecture but trained with a different losses.

These models give different results. I want to «balance» between these models combining their weights lineary:

combined_weights = k * weights_a + (1 - k) * weights_b

The problem I working on allows for this approach. It's argued that this approach is more efficient than interpolating the outputs.

I also have a PyTorch implementation of this approach:

k = 0.3
model_a = torch.load(...)
model_b = torch.load(...)
combined_model = OrderedDict()

for i, v_a in model_a.items():
    v_b = model_b[i]
    combined_model[i] = k * v_a + (1 - k) * v_b

torch.save(combined_model, ...)

How can I do the same in TensorFlow? I've tried to do it combining weights of my models:

k = 0.3
weights_a = model_a.get_weights()
weights_b = model_b.get_weights()

combined_weights = k * weights_a + (1 - k) * weights_b

combined_model.set_weights(combined_weights)

But got an error at k * weights_a:

can't multiply sequence by non-int of type 'float'

So, how can I do it?

tensorflow machine-learning
1 Answers

That is because of the type mismatch. Both weights_a and weights_b are just regular python lists that can't be multiplied by floats (this is not the operation that you are looking anyway).

Weights are stored in ndarray format as elements of those lists. Probably the easiest way to make it work is to loop through the weights and perform the operation for each ndarray

k = 0.3

weights_a = ...
weights_b = ...
combined = []

for i in range(len(weights_a)):
    c = k * weights_a[i] + (1 - k) * weights_b[i]
    combined.append(c)
user_95188900
Related