Huggingface AlBert tokenizer NoneType error with Colab

user_8426105 0
Viewed 6695

I simply tried the sample code from hugging face website: https://huggingface.co/albert-base-v2

`from transformers import AlbertTokenizer, AlbertModel` 
`tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')`
`text = "Replace me by any text you'd like."`
`encoded_input = tokenizer(text, return_tensors='pt')`

then I got the following error at the tokenizer step: ----> 5 encoded_input = tokenizer(text, return_tensors='pt')

TypeError: 'NoneType' object is not callable

I tried the same code on my local machine, it worked no problem. The problem seems within Colab. However, I do need help to run this model on colab GPU.

My python version on colab is Python 3.6.9

huggingface-tokenizers huggingface-transformers google-colaboratory
5 Answers

I found the answer. After install, import the AlbertTokenizer and Tokenizer=..., I received an error asking me to install 'SentencePiece package'. However, after I install this package and run tokenizer again, I started receiving the error above. So I open a brand new colab session, and install everything including the SentencePiece before creating tokenizer, and this time it worked. The Nonetype error simply means it doesn't know what is 'albert-base-v2'. However if you install the packages in right order colab will recognize better the relationship between AlbertTokenizer and SentencePiece. In short for this to work in colab 0. Open a new Colab session 1. Install Transformers and SentencePiece 2. import AlbertTokenizer 3.create tokenizer.

user_84261050

MeiNan Zhu's answer is correct.

Installing or importing SentencePiece before transformers works.

pip install Sentencepiece
!pip install transformers

tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased', do_lower_case=True)

type(tokenizer)

transformers.models.xlnet.tokenization_xlnet.XLNetTokenizer

user_44632640

I just tried restarting the kernel and run again after installing - !pip install sentencepiece - and it worked..

user_184838250

If you have installed transformers and sentencepiece library and still face NoneType error, restart your colab runtime by pressing shortcut key CTRL+M .

(note the dot in shortcuts key)

or use runtime menu and rerun all imports.

Note: don't rerun the library installation cells (cells that contain pip install xxx)

user_84013740

Had the exact same problem, just restarting the Jupyter Notebook kernel after installing the sentencepiece library did the trick for me.

user_96210810
Related