I have a dataframe containing Airbnb listings. It contains some freetext columns, e.g. 'description' 'summary' and 'host info'. I want to combine these freetext columns into one string per row in order to analyse the words. In particular I'd like to find the 10 most frequently occurring words within my whole dataframe (excluding stopwords).
My issue is that some of the listings have used the exact same text for two or more fields. For example, 'description' and 'summary' are occasionally the exact same. Or, the 'host info' has been pasted to the end of 'summary'. I need to remove these duplicates, partially to get the best result, and partly to speed things up - within these columns are 27 million words total so this is really slowing me down! It may make more sense to search for duplicate sentences rather than whole columns, but either would be an improvement.
To concatenate each column within the row I've used:
df['all_description'] = df.name.astype(str) + " " + df.summary.astype(str) + " " + .....
Then to analyse the word content of the whole thing:
import nltk.corpus
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
text_tokens = word_tokenize(' '.join(df['all_description']))
tokens_without_sw = [word for word in text_tokens if not word in stopwords.words()]
Aside from deleting the duplicates, if anyone knows a quicker way to get the top 10 words whilst excluding stopwords I'd be very happy to hear it.