Row merging based on two conditions

Viewed 41

I am currently cleaning my data set for a farm and I need to merge the records from 3 separate rows into one. I'd like the get all the records merged based on columns FARM and SHED.

The dataset currently looks like this: Current state of dataset

Ideally, I wish to make it look like this:

Desired results

What is the best solution to have it cleaned up? I'd like to move columns 'Feed total' and 'Feed per bird' to fill the empty spaces in the rows above. Then I'd like to move around the columns 'Average weight 1A' to row, where the shed number is 1, 'Average weight 2A' to row, where the shed number is 2 etc...

Thanks!

EDIT: Sample dataset attached to give a bigger picture on the dataset

Date Time Age Mortality day Mortality total Feed total Feed per bird Water total Water per bird Farm Shed DT Average weight 1A Average weight 1B Average weight 2A Average weight 2B Average weight 3A Average weight 3B Average weight 4A Average weight 4B Average weight 5A Average weight 6A Average weight 5B Average weight 6B Average Weight 6A1 Average weight 7A Average weight 8A
06/06/2022 23:50:01 -1.0 0.0 0.0 0.0 0.0 182.0 0.004 Park Farm 1 2022-06-06 23:50:00
06/06/2022 23:50:01 -1.0 0.0 0.0 0.0 0.0 105.0 0.003 Park Farm 2 2022-06-06 23:50:00
06/06/2022 23:50:01 -2.0 0.0 0.0 0.0 0.0 4.0 0.0 Park Farm 4 2022-06-06 23:50:00
06/06/2022 23:50:01 -1.0 0.0 0.0 0.0 0.0 82.0 0.003 Park Farm 3 2022-06-06 23:50:00
06/06/2022 23:50:01 Park Farm 1 2022-06-06 23:50:00 2494 0.0 2577.0 0.0 2460.0 0.0 2467.0 1972.0
06/06/2022 23:50:00 4.0 94.0 367.0 600.0 0.018 1473.0 0.044 Green Farm 1 2022-06-06 23:50:00
06/06/2022 23:50:00 4.0 72.0 239.0 340.0 0.01 1480.0 0.044 Green Farm 2 2022-06-06 23:50:00
06/06/2022 23:50:00 4.0 94.0 288.0 630.0 0.019 1443.0 0.043 Green Farm 3 2022-06-06 23:50:00
06/06/2022 23:50:00 3.0 88.0 176.0 290.0 0.009 487.0 0.015 Green Farm 4 2022-06-06 23:50:00
06/06/2022 23:50:00 3.0 105.0 248.0 520.0 0.016 365.0 0.011 Green Farm 5 2022-06-06 23:50:00
06/06/2022 23:50:00 3.0 72.0 168.0 560.0 0.017 289.0 0.009 Green Farm 6 2022-06-06 23:50:00
06/06/2022 23:50:00 Green Farm 1 2022-06-06 23:50:00 1861 1874.0 1990.0 2083.0 1959.0 2160.0
06/06/2022 23:50:01 27.0 40.0 1153.0 11649 0.249 Castle Farm 1 2022-06-06 23:50:00
06/06/2022 23:50:01 26.0 21.0 941.0 11289 0.261 Castle Farm 4 2022-06-06 23:50:00
06/06/2022 23:50:01 26.0 22.0 962.0 10277 0.236 Castle Farm 3 2022-06-06 23:50:00
06/06/2022 23:50:01 27.0 53.0 1004.0 12204 0.259 Castle Farm 2 2022-06-06 23:50:00
06/06/2022 23:50:01 6686.8 0.139 Castle Farm 1 2022-06-06 23:50:00
06/06/2022 23:50:01 6446.9 0.146 Castle Farm 4 2022-06-06 23:50:00
06/06/2022 23:50:01 6031.5 0.135 Castle Farm 3 2022-06-06 23:50:00
06/06/2022 23:50:01 6742.5 0.14 Castle Farm 2 2022-06-06 23:50:00
06/06/2022 23:50:01 Castle Farm 1 2022-06-06 23:50:00 1.508
06/06/2022 23:50:01 Castle Farm 4 2022-06-06 23:50:00 1.542
06/06/2022 23:50:01 Castle Farm 3 2022-06-06 23:50:00 1.493
06/06/2022 23:50:01 Castle Farm 2 2022-06-06 23:50:00 1.542
06/06/2022 23:50:01 24.0 36.0 1211.0 6380.0 0.156 9769.0 0.24 Blue Farm 1 2022-06-06 23:50:00
06/06/2022 23:50:01 24.0 31.0 1317.0 5860.0 0.144 9775.0 0.24 Blue Farm 2 2022-06-06 23:50:00
06/06/2022 23:50:01 24.0 29.0 1431.0 6470.0 0.159 9571.0 0.236 Blue Farm 2 2022-06-06 23:50:00
06/06/2022 23:50:01 24.0 28.0 947.0 5710.0 0.139 9755.0 0.238 Blue Farm 4 2022-06-06 23:50:00
06/06/2022 23:50:01 24.0 29.0 1089.0 6020.0 0.136 10399 0.234 Blue Farm 5 2022-06-06 23:50:00
06/06/2022 23:50:01 24.0 30.0 1489.0 5640.0 0.128 10303 0.234 Blue Farm 6 2022-06-06 23:50:00
06/06/2022 23:50:01 Blue Farm 15 2022-06-06 23:50:00 1289.0 1267.0 1268.0 1243.0
06/06/2022 23:50:01 Blue Farm 5 2022-06-06 23:50:00 1418 1257.0 1462.0 1320.0 1332.0 1309.0 877.0 1248.0
06/06/2022 23:50:38 1.0 0.0 0.0 0.0 0.0 Upper Farm 38 2022-06-06 23:50:00
06/06/2022 23:50:38 1.0 0.0 0.0 0.0 0.0 Upper Farm 3 2022-06-06 23:50:00
06/06/2022 23:50:38 Upper Farm 1 2022-06-06 23:50:00 2.849
06/06/2022 23:50:35 1.0 0.0 0.0 0.0 0.0 34.0 0.001 Seaside Farm 1 2022-06-06 23:50:00
06/06/2022 23:50:35 1.0 0.0 0.0 0.0 0.0 35.0 0.001 Seaside Farm 2 2022-06-06 23:50:00
06/06/2022 23:50:35 1.0 0.0 0.0 0.0 0.0 26.0 0.001 Seaside Farm 3 2022-06-06 23:50:00
06/06/2022 23:50:35 1.0 0.0 0.0 0.0 0.0 35.0 0.001 Seaside Farm 4 2022-06-06 23:50:00
06/06/2022 23:50:35 Seaside Farm 5 2022-06-06 23:50:00 39 40.0 43.0 42.0 45.0 46.0 46.0 46.0
1 Answers

I would start like this, this is assuming you only want one row/record per farm-shed.

import pandas as pd
import functools

# dataframe is df

# First combine horizontally: get the average weight into one column

weight_cols = [col for col in df.columns if col.startswith("Average weight")]
df2 = (
 df.assign(**{
    'Average weight': lambda df: functools.reduce(pd.Series.combine_first, [df[col] for col in weight_cols])
 })
 .drop(columns=weight_cols)
)


# Get first value from each column in each group

grouped_by = ['Farm', 'Shed']
remaining_cols = df2.columns.difference(grouped_by, sort=False)

(df2.groupby(grouped_by)
    .agg({col: "first" for col in remaining_cols})
    .reset_index())

If you want to safeguard against duplicated data, more checks are needed.

Related