Is it possible to find or match two names which has different special characters django

user_2757344 0
Viewed 96

I am trying to populate cities in a state using OpenstreetMaps API to my django application. The database was populated with some cities already. I am facing duplicate data issue since the name in cities sometime has special characters in it.

For example in country Turkey, The state Bursa has city Gursu. My database has a city object with name Gürsu. And the city name from Openstreet Map API is Gürsü. I am trying to find a solution to match existing city with special character name and update it if it exists. So that I can avoid duplicates.

special-characters unicode django localization python
2 Answers

The solution involving unicode is to match text according to UAX#10. You can do that in the database or in Python (possibly using PyICU). Here's some short code demonstrating:

#!/usr/bin/env perl
use 5.010;
use utf8;
use open qw(:std :encoding(UTF-8));
use Unicode::Collate qw();

my $c = Unicode::Collate->new(normalization => undef, level => 1);
my @g = qw(Gursu Gürsu Gursü Gürsü);

for my $o (@g) {
    for my $i (@g) {
        say "$i matches $o" if -1 != $c->index($o, $i, 0);
    }
}

__END__
Gursu matches Gursu
Gürsu matches Gursu
Gursü matches Gursu
Gürsü matches Gursu
Gursu matches Gürsu
Gürsu matches Gürsu
Gursü matches Gürsu
Gürsü matches Gürsu
Gursu matches Gursü
Gürsu matches Gursü
Gursü matches Gursü
Gürsü matches Gursü
Gursu matches Gürsü
Gürsu matches Gürsü
Gursü matches Gürsü
Gürsü matches Gürsü
user_463950

First of all they are not same they have different ASCII values. But if you want to match ü with u You need to work a bit for this you group the similar characters you think in a list like this is rough solution you can modify accordingly

import difflib
similar_groups=[['ü','u']] #add similar special characters here
country = 'Gursu'
country_b = 'Gürsü'
output_list = list(set([li[-1:] for li in difflib.ndiff(country, country_b) if li[0] != ' ']))
match=False #keep false for match found
print(output_list)
for val in similar_groups:
    if(sorted(output_list)==sorted(val)):
        match=True
    else:
        match=False

if match:
    print("Equal")
    #update or skip your stuff here
user_46449420
Related