Multiprocessing with doc containing custom attributes raises serialization error

Viewed 47

I have a spaCy pipeline in which a custom attribute is attached to each doc.

import spacy 

class MyAttr:
  def __init__(self):
    self.elements:List[MySubAttr]

class MySubAttr:
  def __init__(self):
    self.matches: List[Tuple[str, int, int]]
    self.matches2: List[Tuple[str, List[int]]
    self.range: int

class AssignAttr:
  def __init__(self, nlp):
    self.nlp = nlp
  def __call__(self, doc):
    doc._.my_attr = MyAttr()
    return doc

@spacy.language.Language.factory("assign_attr")
def create_multigram_component(
    nlp: spacy.language.Language, name: str
):  
    """create component multigram"""
    return AssignAttr(nlp)

nlp = spacy.load("fr_core_news_lg")
nlp.add_pipe("assign_attr")

Everything is fine when i'm processing the pipeline but if I use nlp.pipe([text, ...], n_process>1) I got an TypeError MyAttr is not serializable. I don't understand why this is happening.

0 Answers
Related