Alternatives to NER taggers for long, heterogeneous phrases?

IMHO NER (or model-based entity extraction in general) alone is a poor choice of methodology for this particular problem as it requires LOTS of manual annotation to do it right. Instead I suggest using Word2Vec (https://radimrehurek.com/gensim/models/word2vec.html) with phrasing (https://radimrehurek.com/gensim/models/phrases.html).

The idea is to have an unsupervised model containing phrases and their similarities which can then queried using some seed words to list all possible ingredients (e.g. "cat" produces similar words like "dog" or "rat"). Next step would be either to create dictionaries containing the ingredient words & phrases or try clustering the vocabulary of the model using cosine similarity between each word/phrase pair.

Now if you want to take things further you can always match your created dictionaries/clusters back to the corpus the W2V model was trained on and then train a custom entity recognition model using those matches as you now have annotated examples.

I believe this is a Multiword-Expression problem.

There are a few ways you can try to solve this:

1. Build a named entity recognition model (NER)
2. Search with Regex for a fixed set of known phrases
3. Chunking tokens with POS tags
4. Find collocations of tokens

Let's look at each of these

Build a named entity recognition model (NER)

Named Entity Recognition labels known spans of tokens an a entity type For each input token you have to label it as part of a known named entity.

Eddy N PERSON
Bonte N PERSON
is V O
woordvoerder N O
van Prep O
diezelfde Pron O
Hogeschool N ORG
. Punc O

This is costly and requires a lot of time for labelling. It is probably not a good choice for your task.

Search with Regex

This is not a bad idea, using some known phrases you could easily search input texts with minimal word boundaries for context.

import re
re.findall(r"\bmay contain nuts\b", text)

This would require you knowing all phrases you want to search for up front, and might not be possible.

Chunking tokens with POS tags

This could be a good intermediate step but could give many false positives.

You could do this my knowing the sequences of POS tags you expect

may     MD
contain VB
nuts    NNS

Then you could use chunking with the known tag sequence (MD, VB, NNS). The problem is that you may not know this, and would have to capture many use cases. It will also capture many sequences which you wont want to capture (False Positive)

Find collocations of tokens

This is probably the best way, as it seems you are looking for a highly common sequences of words (tokens) in a corpus.

You can do this using:

• Word2Vec Phrases
• NLTK Collocations

Both do the same thing, they look for statistically common sequences of tokens which occur in a corpus.

That can then be used to extract the same collocation phrases from new texts.

It looks like your ingredient list is easy to split into a list. In that case you don't really need a sequence tagger; I wouldn't treat this problem as phrase extraction or NER. What I would do is train a classifier on different items in the list to label them as "food" or "non-food". You should be able to start with rules and train a basic classifier using anything really.

Before training a model, an even simpler step would be to run each list item through a PoS tagger (say spaCy), and if there's a verb you can guess that it's not a food item.