tokenize

157 Questions
Newest Active Unanswered Frequent
Score
View
Card Compact
Python find offsets of a word token in a text
user_7588360
•
3
2
1280
stringtokenizer tokenize python
Huggingface error: AttributeError: 'ByteLevelBPETokenizer' object has no attribute 'pad_token_id'
user_30236150
•
2
1
1034
huggingface-tokenizers huggingface-transformers pytorch tokenize python
BERT model : "enable_padding() got an unexpected keyword argument 'max_length'"
user_96212620
•
2
1
2219
bert-language-model padding tokenize nlp
Prevent Spacy tokenizer from splitting on specific character
user_102183120
•
2
1
803
spacy tokenize nlp python
spacy how do I make a matcher which is noun-noun without white space within it?
user_131033240
•
2
1
349
spacy tokenize nlp
Tokenize a sentence where each word contains only letters using RegexTokenizer Scala
user_32798930
•
1
1
407
apache-spark tokenize scala regex
How to untokenize BERT tokens?
user_79730420
•
4
2
3642
huggingface-tokenizers bert-language-model huggingface-transformers tokenize python
Solr wildcard search incorrect result
user_95453970
•
3
3
505
solr tokenize java
How to build a simple tokenizer
user_99696420
•
1
1
111
tokenize substring python
How does the BERT tokenizer result in an input tensor shape of (b, 24, 768)?
user_114075680
•
2
1
716
bert-language-model deep-learning tokenize nlp
Hot Questions