Python: fuzzywuzzy, the output of the first value is correct, the others are NaN

Viewed 109

I'm stuck in a very strange problem: I have two dfs and I have to match strings of one df with the strings of the other df, by similarity. The target column is the name of the television program (program_name_1 & program_name_2). In order to let him choose from a limited set of data, I also used the column 'channel' as filter.

The function applies the fuzzy algorithm and gives as result the match of the elements from the columns program_name_1 with program_name_2 and the score similarity between them.

The really strange thing is that the output works fine just for the first channel, but for all the next channels it doesn't. The first column (scorer_test_2), that just prints the program_name_1 is always correct, but scorer_test_2 (that should print program_name_2) and the similarity columns are NaN.

I did a lot of checks on the dfs: I am sure that the names of the columns are the same of the names in the lists and that in the other channels, there are all the data I'm asking for.

The strangest thing is that the first channel and all the other channels are in the same df, for this reason there are no differences between the data of the channels.

I will show you 'toys dts', to ley you understand better the problem:

df1 = {'Channel': ['1','1','1','2','2','2','3','4'], 'program_name_1': ['party','animals','gucci','the simpson', 'cars', 'mathematics', 'bikes', 'chef']}
df2 = {'Channel': ['1','1','1','2','2','2','3','4'], 'program_name_2': ['parties','gucci_gucci','animal','simpsons', 'math', 'the car', 'bike', 'cooking']}
df1 = pd.DataFrame(df1, columns = ['Channel','program_name_1'])
df2 = pd.DataFrame(df2, columns = ['Channel','program_name_2'])

that will print for the df1:

  Channel program_name_1
       1          party
       1        animals
       1          gucci
       2    the simpson
       2           cars
       2    mathematics
       3          bikes
       4           chef

and for the df2:

  Channel program_name_2
       1        parties
       1    gucci_gucci
       1         animal
       2       simpsons
       2           math
       2        the car
       3           bike
       4        cooking

and here the code:

scorer_test_1 = df_1.loc[(df_1['Channel'] == '1')]['program_name_1']
scorer_test_2 = df_2.loc[(df_2['Channel'] == '1')]['program_name_2']

# creation of a function for the score
def scorer_tester_function(x):
    matching_list = []
    similarity = []
    # iterate on the rows
    for i in scorer_test_1:
        if pd.isnull(i):
            matching_list.append(np.null)
            similarity.append(np.null)
        else:
            ratio = process.extract(i, scorer_test_2, limit=5, scorer=scorer_dict[x])
            matching_list.append(ratio[0][0])
            similarity.append(ratio[0][1])
    my_df = pd.DataFrame()
    my_df['program_name_1'] = scorer_test_1
    my_df['program_name_2'] = pd.Series(matching_list)
    my_df['similarity'] = pd.Series(similarity)

    return my_df

print(scorer_tester_function('R').head())

The output that I would like to get for all the channels, but I just get if I pass the first channel in the code is this:

for the channel[1]:

program_name_1 program_name_2 similarity
    party          parties        95
    animals        animal         95
    gucci        gucci_gucci      75

for the channel[2]:

  program_name_1 program_name_2 similarity
   the simpson     simpsons        85
      cars          the car        75
   mathematics       math          70

This is the output I get if I ask for the channel 2 or next:

code:

scorer_test_1 = df_1.loc[(df_1['Channel'] == '2')]['program_name_1']
scorer_test_2 = df_2.loc[(df_2['Channel'] == '2')]['program_name_2']

output:

  Channel program_name_1 program_name_2 similarity
     2     the simpson        NaN           NaN
     2        cars            NaN           NaN
     2    mathematics         NaN           NaN

I hope someone can help me :)

Thanks!

1 Answers

This was for Index mismatch, resetting indices after adding first dataseries can do the work!

def scorer_tester_function(x):
    matching_list = []
    similarity = []
    # iterate on the rows
    for i in scorer_test_1:
        if pd.isnull(i):
            matching_list.append(np.null)
            similarity.append(np.null)
        else:
            ratio = process.extract(i, scorer_test_2, limit=5)#, scorer=scorer_dict[x])
            matching_list.append(ratio[0][0])
            similarity.append(ratio[0][1])
    my_df = pd.DataFrame()
    my_df['program_name_1'] = scorer_test_1
    print(my_df.index)
    my_df.reset_index(inplace=True)
    print(my_df.index)
    my_df['program_name_2'] = pd.Series(matching_list)
    my_df['similarity'] = pd.Series(similarity)

    return my_df
Related