How to combine 2 dataframe, create a row that appear only in the second dataframe but not in the 1st and groupby to get the sum?

user_18918480 0
Viewed 51

I want to combine 2 dataframes. I have tried several methods but not sure how I can achieve the final dataframe. Appreciate any advice on how can i do this.

data_list_1 = [['Employee', 'Course Name', 'Status'],
              ['Abel', "Course_A", "Completed"],
              ['Bain', "Course_A", "Incomplete"]]

data_list_2 = [['Employee', 'Course Name', 'Lesson Name', 'Lesson Score', 'Status'],
              ['Abel', 'Course_B', 'Lesson_1', 100, ""],
              ['Abel', 'Course_B', 'Lesson_2', 100, ""],
              ['Abel', 'Course_B', 'Lesson_3', 100, ""],
              ['Abel', 'Course_B', 'Lesson_4', 100, ""],
              ['Bain', 'Course_B', 'Lesson_1', 100, ""],
              ['Bain', 'Course_B', 'Lesson_2', 100, ""],
              ['Coot', 'Course_B', 'Lesson_1', 100, ""],
              ['Coot', 'Course_B', 'Lesson_2', 100, ""],
              ['Coot', 'Course_B', 'Lesson_3', 100, ""],
              ['Coot', 'Course_B', 'Lesson_4', 100, ""],
              ['Coot', 'Course_B', 'Lesson_5', 100, ""]]

Course_A_df = pd.DataFrame(data_list_1[1:], columns = data_list_1[0])
Course_B_df = pd.DataFrame(data_list_2[1:], columns = data_list_2[0])

I want to have the following dataframe to use it in Tableau for visualisation purpose. Basically the final df should also have Coot with None values and for Course_B Status to be completed if all 5 Lesson score is 100.

to_achieved = [['Employee', 'Course Name', 'Lesson Name', 'Lesson Score', 'Status'],
              ['Abel', "Course_A", None, None, "Completed"],
              ['Bain', "Course_A", None, None, "Incomplete"],
              ['Coot', "Course_A", None, None, None],              
              ['Abel', 'Course_B', 'Lesson_1', 100, "Incomplete"],
              ['Abel', 'Course_B', 'Lesson_2', 100, "Incomplete"],
              ['Abel', 'Course_B', 'Lesson_3', 100, "Incomplete"],
              ['Abel', 'Course_B', 'Lesson_4', 100, "Incomplete"],
              ['Bain', 'Course_B', 'Lesson_1', 100, "Incomplete"],
              ['Bain', 'Course_B', 'Lesson_2', 100, "Incomplete"],
              ['Coot', 'Course_B', 'Lesson_1', 100, "Completed"],
              ['Coot', 'Course_B', 'Lesson_2', 100, "Completed"],
              ['Coot', 'Course_B', 'Lesson_3', 100, "Completed"],
              ['Coot', 'Course_B', 'Lesson_4', 100, "Completed"],
              ['Coot', 'Course_B', 'Lesson_5', 100, "Completed"]]

to_achieved_df = pd.DataFrame(to_achieved[1:], columns = to_achieved[0])
to_achieved_df

I have tried concat and merge but it doesn't seems to give me what i want.

df_concat = pd.concat([Course_A_df, Course_B_df], axis=0, ignore_index=True)
df_concat

merged = pd.merge(left=Course_A_df, right=Course_B_df, left_on='Employee', right_on='Employee', how='left')
merged

For the calculation of status, i have tried groupby, but is that any way i can check if the value is 500 and update the status?

Thank you!

dataframe group-by python
1 Answers

You can .reindex Course_A_df to add missing Employees:

Course_A_df = (
    Course_A_df.set_index("Employee")
    .reindex(Course_B_df["Employee"].unique())
    .reset_index()
)
Course_A_df["Course Name"] = Course_A_df["Course Name"].ffill().bfill()

Prints:

  Employee Course Name      Status
0     Abel    Course_A   Completed
1     Bain    Course_A  Incomplete
2     Coot    Course_A         NaN

Then add "Status" column to Course_B_df:

Course_B_df["Status"] = Course_B_df.groupby(
    ["Employee", "Course Name"], as_index=False
)["Lesson Score"].transform(
    lambda x: "Complete" if x.sum() == 500 else "Incomplete"
)

Prints:

   Employee Course Name Lesson Name  Lesson Score      Status
0      Abel    Course_B    Lesson_1           100  Incomplete
1      Abel    Course_B    Lesson_2           100  Incomplete
2      Abel    Course_B    Lesson_3           100  Incomplete
3      Abel    Course_B    Lesson_4           100  Incomplete
4      Bain    Course_B    Lesson_1           100  Incomplete
5      Bain    Course_B    Lesson_2           100  Incomplete
6      Coot    Course_B    Lesson_1           100    Complete
7      Coot    Course_B    Lesson_2           100    Complete
8      Coot    Course_B    Lesson_3           100    Complete
9      Coot    Course_B    Lesson_4           100    Complete
10     Coot    Course_B    Lesson_5           100    Complete

and finally .concat the two:

out = pd.concat([Course_A_df, Course_B_df])
print(out[["Employee", "Course Name", "Lesson Name", "Lesson Score", "Status"]])

Prints:

   Employee Course Name Lesson Name  Lesson Score      Status
0      Abel    Course_A         NaN           NaN   Completed
1      Bain    Course_A         NaN           NaN  Incomplete
2      Coot    Course_A         NaN           NaN         NaN
0      Abel    Course_B    Lesson_1         100.0  Incomplete
1      Abel    Course_B    Lesson_2         100.0  Incomplete
2      Abel    Course_B    Lesson_3         100.0  Incomplete
3      Abel    Course_B    Lesson_4         100.0  Incomplete
4      Bain    Course_B    Lesson_1         100.0  Incomplete
5      Bain    Course_B    Lesson_2         100.0  Incomplete
6      Coot    Course_B    Lesson_1         100.0    Complete
7      Coot    Course_B    Lesson_2         100.0    Complete
8      Coot    Course_B    Lesson_3         100.0    Complete
9      Coot    Course_B    Lesson_4         100.0    Complete
10     Coot    Course_B    Lesson_5         100.0    Complete
user_100359850
Related