Can't Import BertTokenizer

user_2355903 0
Viewed 5240

I am attempting to use the BertTokenizer part of the transformers package. First I install as below.

pip install transformers

Which says it succeeds.

When I try to import parts of the package as below I get the following.

 from transformers import BertTokenizer
Traceback (most recent call last):

  File "<ipython-input-2-89505a24ece6>", line 1, in <module>
    from transformers import BertTokenizer

  File "C:\Users\User\anaconda3\lib\site-packages\transformers\__init__.py", line 22, in <module>
    from .integrations import (  # isort:skip

  File "C:\Users\User\anaconda3\lib\site-packages\transformers\integrations.py", line 42, in <module>
    from .trainer_utils import PREFIX_CHECKPOINT_DIR, BestRun  # isort:skip

  File "C:\Users\User\anaconda3\lib\site-packages\transformers\trainer_utils.py", line 10, in <module>
    from .tokenization_utils_base import ExplicitEnum

  File "C:\Users\User\anaconda3\lib\site-packages\transformers\tokenization_utils_base.py", line 31, in <module>
    from tokenizers import AddedToken

  File "C:\Users\User\anaconda3\lib\site-packages\tokenizers\__init__.py", line 17, in <module>
    from .tokenizers import Tokenizer, Encoding, AddedToken

ModuleNotFoundError: No module named 'tokenizers.tokenizers'

The package is detailed here so I think it should be available https://huggingface.co/transformers/model_doc/bert.html

huggingface-transformers python
3 Answers

You could do that:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')

it should work correctly.

Anyway I did a test and doing what you did, but it works for me. I can't reproduce your error. Probably you didn't correctly install the library. Try creating a new environment and installing from scratch.

user_72260800

I had a similar issue. The problem seems to be the way I import the tensor flow library. I changed it from import tensorflow as tf to import tensorflow.

user_152045410

I had same issue. My system is Win10.

First, I pip install transformers==4.10.0,tokenizers==0.10.3, then it can't work.

from transformers import AlbertConfig, AlbertForPreTraining, load_tf_weights_in_albert

File "D:\python-venv\pytorch-venv\lib\site-packages\transformers\tokenization_utils_base.py", line 74, in <module>
    from tokenizers import AddedToken
  File "D:\python-venv\pytorch-venv\lib\site-packages\tokenizers\__init__.py", line 79, in <module>
    from .tokenizers import (
ImportError: DLL load failed:

Then, I pip install transformers==2.11.0,tokenizers==0.7.0, it work. You can try this.

user_168234590
Related