Change default learning rate in SpaCy's Optimizer

user_1612432 0
Viewed 2619

If I want to change the learn_rate for the optimizer, what's the correct way of doing it?

From the definition of begin_training and the default optimizer I am guessing I have to provide a component_cfg param, like this 

optimizer = nlp.begin_training(component_cfg={'learn_rate': 0.01})?

But I do not know if its the correct way, because if I call begin_training like this

optimizer = nlp.begin_training(component_cfg={'dummy_param': 0.01})?

No warning or error is given.

spacy python
3 Answers

To change the spaCy default settings, you need to overwrite the default parameters value.

In your case, to change the learning rate you can do the following:

optimizer.learn_rate = 0.01
user_134616020

I doubt this is the "correct" answer, but this worked for me (so it's more like a note to self :p):

  • I didn't use begin_training() to return an optimizer, because it would just create a default one. I think that you might trick it with environment variables, but I haven't tried
  • instead, I ended up spawning my own optimizer. So my code looks like this:
from thinc.neural.optimizers import Adam
from thinc.neural import Model

# .... other bits of code :)

nlp.begin_training()

# optimizer options. Defaults are changed where commented
ops = Model.ops
learn_rate = 0.001
beta1 = 0.9
beta2 = 0.999
eps = 1e-8
L2 = 1e-6
max_grad_norm = 1.0
optimizer = Adam(ops, learn_rate, L2=L2, beta1=beta1, beta2=beta2, eps=eps)
optimizer.max_grad_norm = max_grad_norm
optimizer.device = ops.device

As you might have guessed, most of the above was copy-pasted from the code which generates the default optimizer in the first place: https://github.com/explosion/spaCy/blob/69e70ffae16700e990d60640f27eb7f980c0ba50/spacy/_ml.py#L49

Which seems to be called from the begin_training() method here: https://github.com/explosion/spaCy/blob/4d4b3b0783bdca38493e27dee2939b3ded735c4e/spacy/language.py#L597 Which is where I got the idea that ops should be thinc.neural.Model.ops

user_107853050
user_117287260
Related