Tokenzing multi words in entire corpus

user_15419071 0
Viewed 62

I'm new to NLTK and was wondering if anyone would be able to help me in solving the following situation:

I'm working on a corpus of new articles and have texts in the following format:

t= "While at a rally in Wilmington, Republican presidential nominee Donald Trump made a clear insinuation that Democratic nominee Hillary Clinton has to keep away from becoming president"

and have following code in place:

from nltk.tokenize import word_tokenize
from nltk.tokenize import MWETokenizer
mwetokenizer = MWETokenizer([('Donald','Trump')], separator=' ')
mwetokenizer.add_mwe(('Hillary','Clinton'))
mwetokenized_sentence = mwetokenizer.tokenize(word_tokenize(t))
print(mwetokenized_sentence)

Which gives the following result:

['While', 'at', 'a', 'rally', 'in', 'Wilmington', ',', 'Republican', 'presidential', 'nominee', 'Donald Trump', 'made', 'a', 'clear', 'insinuation', 'that', 'Democratic', 'nominee', 'Hillary Clinton', 'has', 'to', 'keep', 'away', 'from', 'becoming', 'president']

As we can see the words "Hillary Clinton" and "Donald trump" are together which is as required/expected.

Is there any way that I can figure out similar multi-words in an entire corpus of sentences without looking at individual text?

data-science nltk nlp python
2 Answers

If what you want to merge are proper nouns, then I suggest you to use Named Entity Recognition techniques (check out spacy library and its usage per NER):

import spacy

my_sentence = "While at a rally in Wilmington, Republican presidential nominee Donald Trump made a clear insinuation that Democratic nominee Hilary Clinton has to keep away from becoming president."

nlp = spacy.load('en_core_web_lg') # you have several models to choose from
doc = nlp(my_sentence)
print([(ent, ent.label_) for ent in doc.ents])

$> [(Wilmington, 'GPE'), (Republican, 'NORP'), (Donald Trump, 'PERSON'), (Democratic, 'NORP'), (Hilary Clinton, 'PERSON')]

If are more interested in merging together often occurring sequences of tokens which are likely to convey a meaning on their own (New York City, Department of Justice and similar) than you can use the Phrases class in the gensim library.

Finally, if the entities you're looking for are mainly covered by Wikipedia articles, then you may want to try DBPedia Spotlight. Not only will detect Donal Trump phrase but it will also link it to its Wikipedia page. It works for any type of concept, not just PERSONs.

user_11778680

How about this:

with doc.retokenize() as retokenizer:
    for ent in doc.ents:
        retokenizer.merge(doc[ent.start:ent.end])

In fact, you can use spacy to remove punctuations & stop words, and perform lemmatization too!

parser = spacy.load('de_core_news_sm')
def tokenize(text):
    doc = parser(text)
    with doc.retokenize() as retokenizer:
        for ent in doc.ents:
            retokenizer.merge(doc[ent.start:ent.end], attrs={"LEMMA": ent.text})
    return [x.lemma_ for x in doc if not x.is_punct and not x.is_stop]

Example:

>>> text = "Jochen Schweizer ist eines der interessantesten Unternehmen der Welt, hat den Sitz allerdings nicht in der Schweizerischen Eidgenossenschaft."
>>> print(tokenize(text))
>>> [u'Jochen Schweizer', u'interessant', u'Unternehmen', u'Welt', u'Sitz', u'Schwe
user_61180240
Related