I am currently working on a document AI using the LayoutLM model.I am using below tokenizer as apart of my model but it eliminates the accent from the word which leads my model to predict converted English words from document even if there is a word with accent.I would really appreciate if anyone can suggest me to find a way to replace tokenizer with multillingual support or add some parameters to existing tokenizer with LayoutLM model which does not remove accents .I am aware of LayoutXLM but I am afraid if I have leverage of changing the model to LayoutXLM now after so much training. Thank you so much.
LayoutLM tokenizer CODE( Current Existing Code):
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("microsoft/layoutlm-base-uncased", use_fast=True)
tokenizer.tokenize("Kungälv")
Tokenizer OutPUT: ['kung', '##al', '##v']
Expected Output something like below: LayoutXLMTokenizer tokenizer CODE():
from transformers import LayoutXLMTokenizer
tokenizer = LayoutXLMTokenizer.from_pretrained("microsoft/layoutxlm-base")
tokenizer.tokenize("Kungälv")
Tokenizer OutPUT: ['▁Kung', 'ä', 'lv']