How do I get the words that are not in the lexicon of a CFG grammar?

user_6542968 0
Viewed 746

How to make the program return the list of words not covered by the grammar? For example, consider the code below:

    import nltk
    # Define the cfg grammar.
    grammar = nltk.CFG.fromstring("""
    S -> NP VP
    VP -> V NP
    NP -> det N | N
    V -> "eats" | "drinks"
    N -> "President" | "apple"
    det -> "The" | "a" | "an"
    """)
    sentence = "The President Michel eats banana"

    # Load the grammar into the ChartParser.
    cp = nltk.ChartParser(grammar)

    # Generate and print the parse from the grammar given the sentence tokens.
    for tree in cp.parse(sentence.split()):
        print(tree)

it just displays the error message: ValueError: Grammar does not cover some of the input words: "'Michel','banana'".

However, I want to get these words not covered by the grammar to use them elsewhere in the program.

nltk python
1 Answers

You could use grammar.check_coverage(sentence.split()), but it would raise the same exception with a list of the missing words. However, looking at the source of check_coverage method:

def check_coverage(self, tokens):
    """
    Check whether the grammar rules cover the given list of tokens.
    If not, then raise an exception.

    :type tokens: list(str)
    """
    missing = [tok for tok in tokens
               if not self._lexical_index.get(tok)]
    if missing:
        missing = ', '.join('%r' % (w,) for w in missing)
        raise ValueError("Grammar does not cover some of the "
                         "input words: %r." % missing)

You could write a new function based on their example like:

def get_missing_words(grammar, tokens):
    """
    Find list of missing tokens not covered by grammar
    """
    missing = [tok for tok in tokens
               if not grammar._lexical_index.get(tok)]
    return missing

and use like get_missing_words(grammar, sentence.split()) to get ['Michel', 'banana'] in your example.

user_4575710
Related