spaCy Example object format for SpanCategorizer

Paola 0
Viewed 46

I am having an issue with SpanCategorizer that I believe is due to my Example object format and possible its initialization. Can someone provide a very simple Example object with the correct format? Just an example with two docs and two labels will make it for me. I am not getting how the prediction and the reference should look like. There is a gold standard mentioned in spacy documentation, but it looks out-of-date because the line reference = parse_gold_doc(my_data) doesn't work. Thanks so much for your help!

Here is the code I am using to annotate the docs:

``` phrase_matches = phrase_matcher(doc)
    
    # Initializing SpanGroups
    for label in labels:
        doc.spans[label]=[]
    
    # phrase_matches detection and labeling of spans, and generation of SpanGrups for each doc
    for match_id, start, end in phrase_matches:
            match_label = nlp.vocab.strings[match_id]
            span        = doc[start:end]
            span        = Span(doc, start, end, label = match_label)
            
            # Set up of the SpanGroup for each doc, for the different labels
            doc.spans[match_label].append(span) ```

However spaCy is not recognizing my labels.

initialization spacy
1 Answers

If you want/need to create Example objects directly, the easiest way to do so is to use the function Example.from_dict, which takes a predicted doc and a dict. predicted in this context is a Doc with partial annotations, representing data from previous components. For many use-cases, it can just be a "clean" doc created with nlp.make_doc(text):

from spacy.training import Example
from spacy.lang.en import English

nlp = English()
text = "I like London and Berlin"
span_dict = {"spans": {"my_spans": [(7, 13, "LOC"), (18, 24, "LOC"), (7, 24, "DOUBLE_LOC")]}}
predicted = nlp.make_doc(text)
eg = Example.from_dict(predicted, span_dict)

What this function does, is taking the annotations from the dict and using those to define the gold-standard that is now stored in the Example object eg.

If you print this object (using spaCy >= 3.4.2), you'll see the internal representation of those gold-standard annotations:

{'doc_annotation': {'cats': {}, 'entities': ['O', 'O', 'O', 'O', 'O'], 'spans': {'my_spans': [(7, 13, 'LOC', ''), (18, 24, 'LOC', ''), (7, 24, 'DOUBLE_LOC', '')]}, 'links': {}}, 'token_annotation': {'ORTH': ['I', 'like', 'London', 'and', 'Berlin'], 'SPACY': [True, True, True, True, False], 'TAG': ['', '', '', '', ''], 'LEMMA': ['', '', '', '', ''], 'POS': ['', '', '', '', ''], 'MORPH': ['', '', '', '', ''], 'HEAD': [0, 1, 2, 3, 4], 'DEP': ['', '', '', '', ''], 'SENT_START': [1, 0, 0, 0, 0]}}

PS: the parse_gold_doc function in the docs is just a placeholder/dummy function. We'll clarify that in the docs to avoid confusion!

Sofie VL0
Related