tokenize

157 Questions
Newest Active Unanswered Frequent
Score
View
Card Compact
Why is keras tokenizer applying lowercase() to it's own tokens?
user_164413010
•
2
1
218
data-preprocessing word-embedding keras tokenize
Problems using Spacy tokenizer with special characters
user_80113320
•
2
1
1094
spacy tokenize nlp python
How to tokenize a sentence splitting on spaces, except treat quoted segments as a single token?
user_109149500
•
2
1
81
tokenize javascript
Character level tokenization with special tokens
user_152301800
•
2
0
250
tf.data.dataset recurrent-neural-network tensorflow tokenize nlp
Elasticsearch: custom tokenizer split by words and dots
user_51779630
•
4
2
271
elasticsearch tokenize
disabling fast tokenization in allennlp models
user_93005330
•
1
0
87
allennlp huggingface-transformers tokenize multithreading python
what is the difference between len(tokenizer) and tokenizer.vocab_size
user_118108760
•
4
1
1899
huggingface-tokenizers huggingface-transformers tokenize nlp
Tokenizing a dataframe using Tensorflow and Transformers
user_73963060
•
2
1
1974
huggingface-transformers tensorflow dataframe tokenize python
AttributeError: 'GPT2TokenizerFast' object has no attribute 'max_len'
user_152215340
•
2
2
3664
gpt-2 transformer-model huggingface-tokenizers huggingface-transformers tokenize
UnparsedFlagAccessError: Trying to access flag --preserve_unused_tokens before flags were parsed
user_155744570
•
2
0
685
bert-language-model tokenize python
Hot Questions