huggingface-tokenizers

115 Questions
Newest Active Unanswered Frequent
Score
View
Card Compact
what is the difference between len(tokenizer) and tokenizer.vocab_size
user_118108760
•
4
1
1899
huggingface-tokenizers huggingface-transformers tokenize nlp
huggingface pipeline: bert NER task throws RuntimeError: The size of tensor a (921) must match the size of tensor b (512) at non-singleton dimension 1
user_114944050
•
2
0
913
named-entity-recognition huggingface-tokenizers bert-language-model huggingface-transformers python
How to find back the architecture of a pytorch model having only the weight dictionnary?
user_108357580
•
2
0
157
state-dict huggingface-tokenizers huggingface-transformers pytorch
Huggingface transformer export tokenizer and model
user_25143870
•
2
0
183
huggingface-tokenizers huggingface-transformers
AttributeError: 'GPT2TokenizerFast' object has no attribute 'max_len'
user_152215340
•
2
2
3664
gpt-2 transformer-model huggingface-tokenizers huggingface-transformers tokenize
BERT - Is that needed to add new tokens to be trained in a domain specific environment?
user_79380070
•
2
1
728
huggingface-tokenizers bert-language-model huggingface-transformers nlp
Is there a way to use Huggingface pretrained tokenizer with wordpiece prefix?
user_82860100
•
2
1
757
huggingface-tokenizers
Token indices sequence length is longer than the specified maximum sequence length for this model (651 > 512) with Hugging face sentiment classifier
user_152247780
•
6
1
8811
huggingface-tokenizers huggingface-transformers deep-learning sentiment-analysis nlp
How to download hugging face sentiment-analysis pipeline to use it offline?
user_152247780
•
3
1
1217
huggingface-tokenizers huggingface-transformers deep-learning nlp
PEGASUS pre-training for summarisation tasks
user_98511550
•
2
0
142
summarization transformer-model huggingface-tokenizers huggingface-transformers nlp
Hot Questions