This is the code. I want to remove all the stopwords from the sentence. I still get the word 'i'.
from nltk.corpus import stopwords
stopwords = stopwords.words('english')
en_stops=set(stopwords)
x='I am a good boy. I always pay by debts'
[item.lower().rstrip() for item in x.split() if item not in en_stops]
Output I get:
['i', 'good', 'boy.', 'i', 'always', 'pay', 'debts']
NLTK stop words are all lowercase. So, you need to convert your words to lowercase as well, before doing the membership check. You can change the last line of your code snippet to make it work:
[item.rstrip() for item in x.lower().split() if item not in en_stops]
Update:
As suggested in the comments, for more robustness we can use the in-built tokenizers instead of string.split() to take care of punctuations. In that case the code snippet would look something like this:
import string
from nltk.corpus import stopwords
from nltk import word_tokenize, sent_tokenize
stopwords = stopwords.words('english')
en_stops=set(stopwords)
x = 'I am a good boy. I always pay by debts'
tokenized_sentences = list()
exclusion_set = en_stops.union(string.punctuation)
for sent in sent_tokenize(x):
tokenized_sentences.append([word for word in word_tokenize(sent.lower()) if word not in exclusion_set])
The tokenized senteces would look like this:
[['good', 'boy'], ['always', 'pay', 'debts']]