I am trying to use in a in a parallel way the Stanford NLP library.
I have the next data:
data = "dog, cat, house"
distData = spark.sparkContext.parallelize(data)
Then I import the library and the function is defined to be parallelized
import stanza
stanza.download('es', package='ancora', processors='tokenize,mwt,pos,lemma', verbose=True)
stNLP = stanza.Pipeline(processors='tokenize,mwt,pos,lemma', lang='es', use_gpu=True)
def stanza(x):
return stNLP([x])
After that, I use flatMap:
words = distData.flatMap(stanza)
And there is no error, but when I try to print the information os words, I have an error.
words.take(1)