Match mistyped human names - Postgresql/Python

Viewed 35

Fiddle: http://sqlfiddle.com/#!15/91a68/7

Lets say I have a table as follows:

create table names(
  f_name varchar(255),
  l_name varchar(255),
  some_value varchar(10)
);

insert into names
values ('HAROLD', 'BUCKBERGER', '10'),
('DERICK', 'WILLIAMS', '1'),
('DOUG', 'LASKOWS', '15'),
('JOE', 'TAYLOR', '12');

I get an incremental update next month as follows:

create table new_pull(
  f_name varchar(255),
  l_name varchar(255),
  some_value varchar(10)
);

insert into new_pull
values ('HARALD', 'BUCHBERGER', '1'),
('DERRICK', 'WILLIAM', '11'),
('DOUGLAS', 'LASKOWSKI', '12'),
('JON', 'TAYLOR', '13');

It is known that the data in the updates is not clean and has mistyped names. My goal is to match the names between the 2 tables and update some_value in the first table with the latest one if it exists. Ignore any new rows for now.

So the ideal final table output should look like this:

╔════════════════════╦════════════════════╦═════════════╗
║first_name          ║last_name           ║some_value   ║  
╠════════════════════╬════════════════════╬═════════════╣             
║HAROLD              ║BUCKBERGER          ║1            ║
║DERICK              ║WILLIAMS            ║11           ║
║DOUG                ║LASKOWS             ║12           ║
║JOE                 ║TAYLOR              ║12           ║  
╚════════════════════╩════════════════════╩═════════════╝

One idea I had was to use levenshtein

select n.f_name, n.l_name, np.some_value
from names n
inner join new_pull np
on levenshtein(n.f_name, np.f_name) < 3
and levenshtein(n.l_name, np.l_name) < 3;

This generates the following output: enter image description here

You can see that it erroneously joins JOE with JON although both are different names. Changing the threshold would again miss some rows or join some wrong rows.

Is there any reliable matching algorithm which I can use in postgres to get the results I want? I am open to using any python package as well if its more accurate.

The example I produced here is minimal. There are very annoying typos present in the original data. Happy to clear out confusions if there are any.

Appreciate the help!

1 Answers
Related