I would like to make a merge of two Dask Dataframes so that only the common elements between the 2 result at the end of the merge. I expect, therefore, that the number of observations of the resulting dataframe corresponds to the number of observations of the smaller dataframe. I guessed this was done automatically by the "inner" merge, however, to my surprise, when I perform the command
df_225_228=dask_tls228_part01.merge(df_225_1_2, how="inner", on=["docdb_family_id"]).persist()
I end up with a dataframe that is bigger than the two. Specifically df_225_1_2 (the smaller) has 21'479'056 observations; dask_tls228_part01 has a total of 63'172'938 observations and the resulting dataframe, df_225_228, has 216'868'138 observations.
I am wondering the reason why this is happening. My guess is that there are duplicates somewhere but I fear that a cartesian product of the dataframes is performed. Could you please help me out (even if generally speaking) by performing a merge that takes only the common values between the two dask dataframes (and hence the number of observations of the smaller dataframe or below)?
Thank you in advance.