How to Pickle Spacy Model for Use in PySpark Function

Viewed 412

I am running a spacy matcher model as defined:

from pyspark.sql.functions import udf
from pyspark.sql.types import StringType, ArrayType
import spacy
from spacy.matcher import Matcher

nlp = spacy.load("en_core_web_sm")


def spacy_matcher(text):
    doc = nlp(text)
    matcher = Matcher(nlp.vocab)
    matcher.add("NounChunks", None,  [{"POS": "NOUN", "OP": "+"}] )
    matches = matcher(doc)
    spans = [doc[start:end] for _, start, end in matches]
    return [spacy.util.filter_spans(spans)]

matcher2 = udf(spacy_matcher, ArrayType(StringType()))

When I try to apply this udf to a new column:

test = reviews.withColumn('chunk',matcher2('SENTENCE'))
test.show()

I get a pickling error:

NotImplementedError: [E112] Pickling a span is not supported, because spans are only views of the parent Doc and can't exist on their own. A pickled span would always have to include its Doc and Vocab, which has practically no advantage over pickling the parent Doc directly. So instead of pickling the span, pickle the Doc it belongs to or use Span.as_doc to convert the span to a standalone Doc object.

I haven't done a lot with pickling and am unsure how to handle this because I do want to keep the spans because those are what define my chunks. Any idea how to properly pickle?

0 Answers
Related