For example: I have to find out which char in my string is non ascii and convert that to same ascii char.
exa: 'DAÝ' to 'DAY'.
For example: I have to find out which char in my string is non ascii and convert that to same ascii char.
exa: 'DAÝ' to 'DAY'.
Your question has two parts:
1- how to find non-ASCII characters?
For the first part you can use the ord() function to get the code and it should be less than 128 or you can use the following regex:
import re
text = 'DAÝ'
print(re.search(r'[^\x00-\x7F]+', text))
Output:
<re.Match object; span=(2, 3), match='Ý'>
2- how to replace them with a relevant character?
For the second part, the answer is not obvious. You should make a mapping between non-ASCII characters to ASCII characters (manually) or use existing Unicode categories to filter them out (like the other answer on this question). The following code only removes the characters that are in "Mark, nonspacing" character set in Unicode (including accents):
import unicodedata
unicode_text = 'DAÝ£'
filtered_text = ''.join(char for char in unicodedata.normalize('NFD', unicode_text) if unicodedata.category(char) != 'Mn')
print(filtered_text)
output:
DAY£
You can see these characters here: https://www.fileformat.info/info/unicode/category/Mn/list.htm
However, this code does not filter thousands of other Unicode characters like ‡, £, etc. In order to find the right category for your problem, review the existing list of Unicode categories to decide which categories you want to keep: https://www.fileformat.info/info/unicode/category/index.htm
Note: you should first normalize your Unicode characters to make sure you decompose compositions like accents on characters.
Use following code to remove accents from string:
import unicodedata
unicode_text = "DAÝ"
ascii_text = ''.join(char for char in unicodedata.normalize('NFD', unicode_text) if unicodedata.category(char) != 'Mn')
print(ascii_text) # -> "DAY"
This also does work with Zalgo text:
import unicodedata
unicode_text = "S̴̢̲̓o̴̢͒́̇ͅM̴̬̈́̉̽ȩ̴̳́ ̷̲̤͈͐͘̚F̶͔̌̊̾ů̸͈͓̗̌̚n̶͉͚̈́̄K̸̛̗̕y̷̭̍̅ ̶̪̯͇̐T̴̲́̋̚ḛ̸̛̾͑X̵͍͑Ț̷̱͙̄̕"
ascii_text = ''.join(char for char in unicodedata.normalize('NFD', unicode_text) if unicodedata.category(char) != 'Mn')
print(ascii_text) # -> "SoMe FunKy TeXT"