Stanford CoreNLP output is very slow in Python

user_14322271 0
Viewed 164

I'm using NLTK's StanfordDependencyParser to generate dependency trees. Here is the code

cpath = "path to stanford-corenlp-4.2.0-models-english.jar" + os.pathsep + "path to stanford-parser.jar"

if cpath not in os.environ['CLASSPATH']:
     os.environ['CLASSPATH'] = cpath + os.pathsep + os.environ['CLASSPATH']

# TODO: DEPRECATED
# self.dependency_parser_instance_corenlp = StanfordDependencyParser(path_to_models_jar="path to stanford-corenlp-4.2.0-models-english.jar", encoding='utf8')

dependencies = [list(parse.triples()) for parse in self.dependency_parser_instance_corenlp.raw_parse(query)]

# Encode every string in tree to utf8 so string matching will work
for dependency in dependencies[0]:
    dependency[0][0].encode('utf-8')
    dependency[0][1].encode('utf-8')
    dependency[1].encode('utf-8')
    dependency[2][0].encode('utf-8')
    dependency[2][1].encode('utf-8')

For a sentence with 10 words, it takes around 1.5 seconds to generate the output. Is this expected? Can you please steps to improve speed? I've already tried using SR parser and removing all the extra folders(like coref, lexparser, ner, tagger) from the models JAR.

stanford-nlp nltk python
1 Answers

Yes, the (old NLTK implementation) StanfordDependencyParser is very slow. You should not use it. You should use the classes and methods in the nltk.parse.corenlp module, which are much faster.

For more info, see the notes in the CoreNLP documentation or this tutorial.

user_2350190
Related