Library Stanford nlp in pyspark

Viewed 82

I am trying to use in a in a parallel way the Stanford NLP library.

I have the next data:

data = "dog, cat, house"

distData = spark.sparkContext.parallelize(data)

Then I import the library and the function is defined to be parallelized

import stanza
stanza.download('es', package='ancora', processors='tokenize,mwt,pos,lemma', verbose=True)
stNLP = stanza.Pipeline(processors='tokenize,mwt,pos,lemma', lang='es', use_gpu=True)

def stanza(x):
   return stNLP([x])

After that, I use flatMap:

words = distData.flatMap(stanza)

And there is no error, but when I try to print the information os words, I have an error.

words.take(1)

0 Answers
Related