NLTK - distinguishing between colors and words using context

Viewed 187

I'm writing a program to analyze the usage of color in text. I want to search for color words such as "apricot" or "orange". For example, an author might write "the apricot sundress billowed in the wind." However, I want to only count the apricots/oranges that actually describe color, not something like "I ate an apricot" or "I drank orange juice."

Is there anyway to do this, perhaps using context() in NLTK?

1 Answers

Welcome to the broad field of homonymy, polysemy and WSD. In corpus linguistics, this is an approach where collocations e.g. and are used to determine a probability of the juice having the colour "orange" or being made of the respective fruit. Both probabilities are high, but the probability of "jacket" being made of the respective fruit should be much lower. There are different methods to be used. You could ask corpus annotators (specialists, crowdsourcing etc.) to annotate data in a text, which you can use to train your (machine learning) model, in this case a simple classifier. Otherwise you could use large text data to gather collocation counts in combinition with Wordnet, which may give you semantic information whether it is usual for a jacket to be made of fruits. A fortunate detail is that only rarely people use stereotypical colours in text, so you don't have to care for cases like "the yellow banana". Shallow parsing may also help, since colour adjectives should be preferrably used in attributive position. A different approach would be to use word similarity measures (vector space semantics) or embeddings for Word Sense disambiguation (WSD). Maybe this helps: https://web.stanford.edu/~jurafsky/slp3/slides/Chapter18.wsd.pdf https://towardsdatascience.com/a-simple-word-sense-disambiguation-application-3ca645c56357

Related