Get all possible part-of-speech tags for a word using NLTK

Viewed 1396

Some words can have more than one possible part of speech (pos) tag. E.g. 'Stick' is both a noun and a verb.

The pos taggers in NLTK try to guess the correct tag based on context and return only the 1 guess. How can I instead get a list of all possible tags for any given word?

1 Answers

TL;DR

No, not for the default pos_tag function.


In Long

For the default pos_tag function it's not possible.

The default pos_tag function is from the AveragedPerceptron object that uses the predict() function to get the most probable tag: https://github.com/nltk/nltk/blob/develop/nltk/tag/perceptron.py#L48

The function returns the argmax from the list of probable tags:

def predict(self, features):
    '''Dot-product the features and current weights and return the best label.'''
    scores = defaultdict(float)
    for feat, value in features.items():
        if feat not in self.weights or value == 0:
            continue
        weights = self.weights[feat]
        for label, weight in weights.items():
            scores[label] += value * weight
    # Do a secondary alphabetic sort, for stability
    return max(self.classes, key=lambda label: (scores[label], label))

Effectively, if you change the code you code get the score of every possible tag by letting it return self.classes.

But because the features used in tag() requires the previous two tags as features https://github.com/nltk/nltk/blob/develop/nltk/tag/perceptron.py#L156

def tag(self, tokens):
    '''
    Tag tokenized sentences.
    :params tokens: list of word
    :type tokens: list(str)
    '''
    prev, prev2 = self.START
    output = []

    context = self.START + [self.normalize(w) for w in tokens] + self.END
    for i, word in enumerate(tokens):
        tag = self.tagdict.get(word)
        if not tag:
            features = self._get_features(i, word, context, prev, prev2)
            tag = self.model.predict(features)
        output.append((word, tag))
        prev2 = prev
        prev = tag

    return output

The task of returning n-best tags would have to change the simple one-best "greedy" nature of the tagger to something that requires a beam.

Related