I am using vowpalwabbit for a contextual bandit problem. I want to use the cover option as explained here.
I am facing 2 issues with this:
- once the learning phase is over and I use the vw model to make predictions, these predictions (in this case, pmf over the actions) are not stable
- if I save the model and reload it to memory, the predictions are different
Here is an example (using the python wrapper of VW):
import vowpalwabbit.pyvw as pyvw
data_train = ["1:0:0.5 |features a b", "2:-1:0.5 |features a c", "2:0:0.5 |features b c",
"1:-2:0.5 |features b d", "2:0:0.5 |features a d", "1:0:0.5 |features a c d",
"1:-1:0.5 |features a c", "2:-1:0.5 |features a c"]
data_test = ["|features a b", "|features a b"]
model1 = pyvw.vw(cb_explore=2, cover=10)
for data in data_train:
model1.learn(data)
model1.save("saved_model.model")
model2 = pyvw.vw(cb_explore=2, cover=10, i="saved_model.model")
for data in data_test:
print(data)
print(model1.predict(data))
print(model2.predict(data))
I get the following output:
|features a b
[0.75, 0.25]
[0.5, 0.5]
|features a b
[0.7642977237701416, 0.2357022762298584]
[0.5, 0.5]
As you can see, predictions for model 1 are changing (slightly) while predictions for model 2 (which should be the same as model 1) are different.
If I replace cover with bag, I do not get this problem. What is the explanation for this, and is there a way to fix it in VW?