How to tokenize python code using the Tokenize module?

import tokenize
import io

inp = """import nltk
 from nltk.stem import PorterStemmer
 porter_stemmer=PorterStemmer()
 words=["connect","connected","connection","connections","connects"]
 stemmed_words=[porter_stemmer.stem(word) for word in words]
 stemmed_words"""

for token in tokenize.generate_tokens(io.StringIO(inp).readline):
 print(token)

tokenize.tokenize takes a method not a string. The method should be a readline method from an IO object. In addition, tokenize.tokenize expects the readline method to return bytes, you can use tokenize.generate_tokens instead to use a readline method that returns strings.

Your input should also be in a docstring, as it is multiple lines long.

See io.TextIOBase, tokenize.generate_tokens for more info.

If you want to stick with tokenize.tokenize(), then this is what you can do:

from tokenize import tokenize
from io import BytesIO

code = """import nltk
 from nltk.stem import PorterStemmer
 porter_stemmer=PorterStemmer()
 words=["connect","connected","connection","connections","connects"]
 stemmed_words=[porter_stemmer.stem(word) for word in words]
 stemmed_words"""
 
for tok in tokenize(BytesIO(code.encode('utf-8')).readline):
    print(f"Type: {tok.type}\nString: {tok.string}\nStart: {tok.start}\nEnd: {tok.end}\nLine: {tok.line.strip()}\n======\n")

From the documentation you can see:

The generator produces 5-tuples with these members: the token type; the token string; a 2-tuple (srow, scol) of ints specifying the row and column where the token begins in the source; a 2-tuple (erow, ecol) of ints specifying the row and column where the token ends in the source; and the line on which the token was found. The line passed (the last tuple item) is the physical line. The 5 tuple is returned as a named tuple with the field names: type string start end line.

The returned named tuple has an additional property named exact_type that contains the exact operator type for OP tokens. For all other token types exact_type equals the named tuple type field.