I am doing a comparison from a csv to a dictionary and right now it only does an exact match. I am looking to do a partial match so I can recommend the correct field name.
This code is reading 6 different columns and comparing the contents in the columns to a dict of standard values. Currently it put an error message out that says "hotel is invalid check column industry and replace with a standard value." I would like it to say something more along the lines of "hotel is invalid check column industry and considering replacing with hospitality/entertainment" The recommendation I would like to come from a closeness ratio when doing a partial string match and if the ratio is below a certain percent to not include a recommendation to try to keep the recommendation as accurate as possible.
#json file contain dict of stand values
with open('valid.json', 'r') as validvals:
valid = json.load(validvals)with open('valid.json', 'r') as validvals:
valid = json.load(validvals)
df1= df[["Industry",'System Type','Account Type', 'SME Vertical','Country','State/Province']]
mask = df1.apply(lambda c: c.isin(valid[c.name]))
df1.mask(mask|df1.eq(' ')).stack()
for err_i, (r, v) in enumerate(df1.mask(mask|df1.eq(' ')).stack().iteritems()):
errors[filename][err_i] = {"row": r[0],
"column": r[1],
"message": v + " is invalid check column " + r[1] + ' and replace with a standard value'}
Here is the basic set up of my code. Anyone have recommendations on what packages or methods to look at to complete this task? I have looked at FuzzyWuzzy, Regex, and methods like Levenshtein Distance. If anyone has more ideas of packages or methods to best complete this that would be amazing.