tokenize

157 Questions
Newest Active Unanswered Frequent
Score
View
Card Compact
How do I tokenize to separate apostrophes?
user_170068270
•
1
2
32
python-3.8 tokenize python
How does Byte-pair Encoding handle equally frequent pairs?
user_151976060
•
1
0
26
huggingface-tokenizers machine-learning tokenize nlp
How can I prevent the benepar parser from splitting a specific substring when parsing a string?
user_3958570
•
1
1
39
benepar parse-tree tokenize nlp python
Delete brackets from column values
user_174659010
•
1
3
46
pandas dataframe tokenize python-3.x python
I use the word tokenize function on my dataframe, by writing word_dict, but after executing the error message 'expected string or bytes-like object'
user_182650720
•
1
1
25
jupyter-notebook dataframe tokenize python
Is there a simpler way to count the number of tokens in a string with duplicated delimiters in Kotlin?
user_136361210
•
1
1
34
kotlin word-count tokenize regex string
I set 'truncation=True' when using Tapas Tokenizer in transformers, but still get a warning of not setting truncation explicitly
user_196878370
•
2
0
27
huggingface-transformers pytorch tokenize nlp
Slow and Fast tokenizer gives different outputs(sentencepiece tokenization)
user_178110170
•
1
0
63
sentencepiece huggingface-tokenizers tokenize nlp
Equivalent to tokenizer() in Transformers 2.5.0?
user_155609080
•
1
1
67
huggingface-tokenizers bert-language-model huggingface-transformers pytorch tokenize
How do we generate the first target words in machine translation?
user_196146940
•
1
1
24
machine-translation tokenize
Hot Questions