Pandas: How to concat or merge two incomplete dataframe into one more complete dataframe

user_16446640 0
Viewed 347

I would like to concatenate two incomplete data frame with the same data (in theory) regarding a similar index. I tried with pd.concat but I don't managed to get what I need.

Here is a simple example of what I would like to do :

     df1 = pd.DataFrame(
    {
        "A": ["A0", "A1", "A2", "A3"],
        "B": ["B0", "B1", "B2", "B4"],
        "C": ["C0", "C1", "C2", "B5"],
        "D": [np.nan,np.nan,np.nan,np.nan,]
    },
    index=[0, 1, 2, 3],)

df2 = pd.DataFrame(
    {
        "A": ["A0", "A1", "A5", "A6"],
        "B": ["B0", "B1", "B5", "B6"],
        "C": [np.nan,np.nan,np.nan,np.nan,],
        "D": ["D0", "D1", "D5", "D6"],
    },
    index=[0, 1, 5, 6]
)

res_expected = pd.DataFrame(
    {
        "A": ["A0",  "A1",  "A2",  "A3", "A5", "A6"],
        "B": ["B0",  "B1",  "B2",  "B3",  "B5", "B6"],
        "C": ["C0",  "C1",  "C2",  "B5",np.nan,np.nan,],
        "D": ["D0",  "D1",  np.nan,np.nan,"D5", "D6"],
    },
    index=[0, 1, 2, 3, 5, 6]
)

Does someone have an idea ?

Thanks !

pandas outer-join concatenation merge python
2 Answers

You can use combine_first(), as follows:

df_result = df1.combine_first(df2)

combine_first() works as follows:

Combine two DataFrame objects by filling null values in one DataFrame with non-null values from other DataFrame. The row and column indexes of the resulting DataFrame will be the union of the two.

Result:

print(df_result)

    A   B    C    D
0  A0  B0   C0   D0
1  A1  B1   C1   D1
2  A2  B2   C2  NaN
3  A3  B4   B5  NaN
5  A5  B5  NaN   D5
6  A6  B6  NaN   D6
user_150706970 
res_expected=df1.append(df2,ignore_index=True)

This should work

user_109702570
Related