Explanation/interpretation of the parameters in the spaCy config file

user_6475377 0
Viewed 763

I have a couple of questions regarding the parameters that we define in the config.cfg file. Although spaCy's docs do try to explain them, I feel that the explanation isn't really descriptive enough and that lots of things are scattered around the docs, making it difficult to find exactly what you need, especially with spaCy v3, (unless I'm looking at wrong parts of the website) which is recent and hence has really less question/answers in the forums. I'm basically building a Named Entity Recognition (NER) model along with a transformer component. My questions are as follows:

  1. In the following part (same question for corpora.train also), what is the difference between max_length and limit?

    For max_length the docs say "Limitations on training document length"
    For limit, the docs say "Limitation on number of training examples"

    Aren't they both more or less the same thing? I mean I can limit the number of training examples by limiting the document's length itself, right?

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null
  1. In the below snippet, what is the meaning of one 'step'? I understand max_steps=0 means infinite steps. But how do I know how many such 'steps' make one epoch? Also how many example sentences are covered in 1 such step?
[training]
train_corpus = "corpora.train"
dev_corpus = "corpora.dev"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 10
max_steps = 0
eval_frequency = 200
frozen_components = []
before_to_disk = null
  1. How exactly is the learn_rate being modified in the below snippet of code, during the training process? More specifically, what do total_steps and warmup_steps mean?
[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 200
initial_rate = 0.00005
  1. Finally, in the CLI output of the training process, What exactly is this '#'? It was mentioned in one of GitHub discussions that "The # column is the number of optimization steps (= batches processed)" , but what exactly is this 1 batch or 'optimization step'? If the training process shows me the scores for after 200 such 'batches' how do I interpret it (as in how many example sentences have been processed till that point)?
spacy-3 named-entity-recognition spacy nlp python
1 Answers

In the following part (same question for corpora.train also), what is the difference between max_length and limit? For max_length the docs say "Limitations on training document length" For limit, the docs say "Limitation on number of training examples" Aren't they both more or less the same thing? I mean I can limit the number of training examples by limiting the document's length itself, right?

These are different things, you seem to be confused about what a "document" is. You can think of a "doc" as being a single object in spaCy. Different docs don't know anything about each other. A doc is based on a single string. Using normal Python strings as an example:

["cat", "dog", "fish"] # this is three strings
["cat dog fish"] # this is one string

You can see that "take three strings from the list" and "take strings not more than three characters long" are very different things. The values in spaCy are like that.

In the below snippet, what is the meaning of one 'step'? I understand max_steps=0 means infinite steps. But how do I know how many such 'steps' make one epoch? Also how many example sentences are covered in 1 such step?

A "step" is a "batch". A "batch" is running training over some number of examples and updating the model weights once. You can control the size of a batch so it can be any number of examples. An "epoch" is how long it takes the training to see every example once, so if you have 5 documents per batch and 30 training documents then 6 steps would be one epoch.

spaCy doesn't necessarily know anything about "sentences" in training, docs are the basic unit of a batch. Your training examples might all be single sentences but that's not a requirement.

These terms are not spaCy-specific, they are widely used in machine learning.

How exactly is the learn_rate being modified in the below snippet of code, during the training process? More specifically, what do total_steps and warmup_steps mean?

This is from Thinc, see the docs there.

To quote:

Generate a series, starting from an initial rate, and then with a warmup period, and then a linear decline. Used for learning rates.

At the end of total_steps the learning rate stops changing.

Finally, in the CLI output of the training process, What exactly is this '#'? It was mentioned in one of GitHub discussions that "The # column is the number of optimization steps (= batches processed)" , but what exactly is this 1 batch or 'optimization step'? If the training process shows me the scores for after 200 such 'batches' how do I interpret it (as in how many example sentences have been processed till that point)?

A step is the same thing as in #2, it's one batch. Batch size is expressed in docs, not in sentences.

user_3557150
Related