Can I use regex to find typos or alternate spellings of a person's name when searching a rest API databse?

user_19258943 0
Viewed 55

Using the senate.gov website's Lobbying Disclosure Act (LDA) API, I was able to create a database of each individual donation given by lobbyists or organizations that lobby to specific candidates for congressional office.

However, the government's LDA data is rather unorganized as the lobbyists who fill out the forms will often spell the politicians names' incorrectly or use alternate spellings.

Ex: Lobbyists could be donating to the same candidate but could write the payee as John Smith, Jonathan Smith, Jon Smith, etc.

So I am attempting to use regular expressions to ensure my Python program doesn't miss any typos or alternate spellings...

Here is what I had before that worked but didn't account for alternate spellings (I don't want to manually input John Smith, Jonathan Smith, Jon Smith each time, I'd rather just use regex to do: J.*n Smith)

import requests
import json
import csv

Candidate = John Smith

Payee_Parameter = {
    "contribution_payee": Candidate,
    "dt_posted": "ascending",
    "key": "1234"
}


ContributionsLink = "https://lda.senate.gov/api/v1/contributions/"
response = requests.get(ContributionsLink, params=Payee_Parameter)

data = json.loads(response.text)
lines = json.dumps(response.json(),indent=4)

#The rest of my code after this just organizes the results by category into csv file

Now here is my thought process to account for alt spellings and typos with regex. It probably did not work because I don't think you can pass regex into url query parameters.

import requests
import json
import csv
import re

Candidate = r'J.*n Smith'

pattern = re.compile(Candidate)

Payee_Parameter = {
    "contribution_payee": pattern,
    "dt_posted": "ascending",
    "key": "1234"
}

ContributionsLink = "https://lda.senate.gov/api/v1/contributions/"
response = requests.get(ContributionsLink, params=Payee_Parameter)
data = json.loads(response.text)
lines = json.dumps(response.json(),indent=4)



# matches = pattern.finditer(data) <-- Maybe this can come in handy? I am new to regex and decent with Python.

Any ideas on how I can account for those alternate spellings/typos with the LDA API? Thank you!

html python json regex
1 Answers

I think the correct approach would be fuzzy matching. It vectorizes the character string and then compares it's distance using an algorithm. Ideal if you misspell random characters.

Take a look at this: https://towardsdatascience.com/fuzzy-string-matching-in-python-68f240d910fe

It would work like this:

Install fuzzywuzzy

pip install fuzzywuzzy

Run this:

from fuzzywuzzy import fuzz
from fuzzywuzzy import process


print(fuzz.ratio("John Doe","Joe Dow"))
print(fuzz.ratio("John Doe","John M. Doe"))
print(fuzz.ratio("John Doe","Billy Jean"))

Output:

67
84
22
user_122229090
Related