After comparison of Data Frames the output CSV isn't returning all rows

user_5726195 0
Viewed 197

I have found a function that will compare two dataframes. On use it seems to compare them correctly and it is using an outer join, but the output is only 40 rows when it should be 180. What is even more odd is that the index row indexes remain. It starts at row 0 and then skips to row 140.

I have also tried printing the output to check if there is an error during the "to_csv" function and it outputs the same thing.

Function:

def df_difference(df1, df2, which=None):
    # Find rows which are different between two DataFrames.
    comparison_df = df1.merge(df2,
                              indicator=True,
                              how='outer')
    if which is None:
        diff_df = comparison_df[comparison_df['_merge'] != 'both']
    else:
        diff_df = comparison_df[comparison_df['_merge'] == which]
    diff_df.to_csv('data/diff ' + now.strftime('%m-%d-%Y') + '.csv')
    return diff_df

Use of function:

df_compared = df_difference(df_check[['SS', 'F_DOD']], df_to_add_pc[['SS', 'F_DOD']])

Current output:

,    SS,       EnteredDate,  F_DOD       ,_merge
0,   12,      2020-06-18    ,2020-06-01  ,left_only
140, 123,,    2020-05-19                 ,right_only
141, 1234,,   2020-06-06                 ,right_only
142, 12345,,  2020-06-06                 ,right_only
143, 123456,, 2020-05-02                 ,right_only
144, 11234,,  2020-05-23                 ,right_only
145, 22134,,  2020-05-29                 ,right_only
146, 123124,, 2020-05-22                 ,right_only
147, 1234512,,2020-05-28                 ,right_only
148, 521312,, 2020-05-17                 ,right_only
149, 123412,, 2020-05-26                 ,right_only
150, 1236231,,2020-05-25                 ,right_only
151, 782123,, 2020-05-27                 ,right_only
152, 7812312,,2020-05-17                 ,right_only
153, 879122,, 2020-05-10                 ,right_only
154, 981293,, 2020-06-05                 ,right_only
155, 98712,,  2020-05-15                 ,right_only
156, 867891,, 2020-05-11                 ,right_only
157, 12312,,  2020-05-13                 ,right_only
158, 123541,, 2020-05-29                 ,right_only
159, 98712,,  2020-05-09                 ,right_only
160, 908123,, 2020-06-04                 ,right_only
161, 897291,, 2020-06-04                 ,right_only
162, 8791231,,2020-06-01                 ,right_only
163, 89712,,  2020-05-12                 ,right_only
164, 9081,,   2020-05-17                 ,right_only
165, 98712,,  2020-05-30                 ,right_only
166, 0123,,   2020-06-06                 ,right_only
167, 8213,,   2020-05-22                 ,right_only
168, 9891,,   2020-05-19                 ,right_only
169, 90812,,  2020-05-24                 ,right_only
170, 908122,, 2020-06-01                 ,right_only
171, 52131,,  2020-05-23                 ,right_only
172, 34512,,  2020-05-09                 ,right_only
173, 987891,, 2020-04-19                 ,right_only
174, 908102,, 2020-05-05                 ,right_only
175, 09521,,  2020-05-08                 ,right_only
176, 82134,,  2020-05-16                 ,right_only
177, 87182,,  2020-05-10                 ,right_only
178, 76812,,  2020-05-28                 ,right_only
179, 78921,,  2020-04-18                 ,right_only
180, 89612,,  2020-04-17                 ,right_only
181, 786123,, 2019-10-04                 ,right_only

Expected output: Looking for the same format as the current output but including the 100+ rows that are missing.

pandas python
2 Answers

For viewing the entire output, Just return the merged output without filtering:

def df_difference(df1, df2, which=None):
    # Find rows which are different between two DataFrames.
    comparison_df = df1.merge(df2,indicator=True,how='outer')
    return comparison_df
user_126903440

You can try this as well.

def df_difference(df1, df2, which=None):
    for col in d2.columns.value:
        d1[col] = d2[col]
        return d1
user_87445760
Related