dataframe: transform row-based transaction data into aggregates per date

Viewed 347

I retrieve data from a SQLITE Database (and transform it to a pandas dataframe) in the following format:

Driver | Date loading | Date unloading | Loading Adress | Unloading Address
Peter  | 02.05.2020   | 03.05.2020     | 12342, Berlin  | 14221, Utrecht
Peter  | 03.05.2020   | 04.05.2020     | 14221, Utrecht | 13222, Amsterdam
Franz  | 03.05.2020   | 03.05.2020     | 11111, Somewher| 11221, Somewhere2
Franz  | 03.05.2020   | 05.05.2020     | 11223, Upsalla | 14231, Berlin

The date range can be specified for the query, so that it gives an overview over which driver has which transports to deliver within the specified date range, ordered by date.

The goal of the transformation I want to do is a weekly plan for each driver, with the dates from the range sorted in the available columns. So for the data above, this would look like the following:

Driver | 02.05.2020           | 03.05.2020            | 04.05.2020         | 05.05.2020      |
Peter  | Loading:             | Unloading:              Unloading:
         12342, Berlin          14221, Utrecht          13222, Amsterdam
                                Loading:
                                14221, Utrecht

Franz  |                      | Loading:              |                    | Unloading:
                                11111, Somewher                              14231, Berlin
                                Unloading:
                                11221, Somewhere2
                                Loading:
                                11223, Upsalla

Is there any way to achieve the described output with dataframe operations? Within the single data columns I will need to keep the order, which is loading first, unloading second, and then go to the next data row if the date is the same.

2 Answers

I will write my pseudoish solution, where actually it is a solution, it only lacks a single entity, task_id, I will elaborate this later. I will call your dateframe (1st one in question) df and I create a transformed version as t_df. This t_df will be an unpivoted table where dates and addresses are unified.

I will create a data frame as follows:

Driver | Date         | Task       | Address 
Peter  | 02.05.2020   | Loading    | 12342, Berlin
Peter  | 03.05.2020   | Unloading  | 14221, Utrecht

with this unpivot data frame I can now pivot it as you wanted Like a schedule.

m,n = df.shape
t_df = pd.DataFrame(columns=['driver', 'date', 'task', 'address'])
t_df['Driver'] = df['Driver'].tolist() * 2
t_df['Date'] = df['Date loading'].tolist() + df['Date unloading'].tolist()
t_df['Address'] = df['Loading Address'].tolist() + df['Unloading Address'].tolist()
t_df['Task'] = ['Loading'] * m + ['Unloading'] * m

Now, I add the values task + address as a column.

t_df['Compound'] = t_df[['Task', 'Address']].agg(': '.join, axis=1)

concat_array = lambda x: '; '.join(x)

schedule = pd.crosstab(index=t_df['Driver'], columns=t_df['Date'], values=t_df['Compound'],
    aggfunc=concat_array)

I will get the following data frame:

Date                02.05.2020  ...                05.05.2020
Driver                          ...                          
Franz                      NaN  ...  Unloading: 14231, Berlin
Peter   Loading: 12342, Berlin  ...                       NaN

Now, as I said at the very beginning of the answer, you need some kind of task identifier to match which task belongs to which assuming there are multiple load & unload operations in the same day. You need to assign some kind of task_id and then put it in the Compound column.

Note: I used '; ' for separation of the tasks, you may want to use something else.

You can check complete code file in gist.

I would recommend taking advantage of Pandas's multiindexing in order to organize and sort your data. Rather than having separate columns for 'Date loading', 'Date unloading', 'Loading Address', 'Unloading Address', I'm going to replace those with one column for 'Date', one for 'Address', and a new column 'Loading', so we can sort the data more easily. I also added a delivery_id column to keep the keep the loading and unloading pairs matched up. So in the first step I'm just reorganizing the data into a more sortable dataframe:

data = [['Peter', '02.05.2020', '03.05.2020', '12342, Berlin', '14221, Utrecht'],
        ['Peter', '03.05.2020', '04.05.2020', '14221, Utrecht', '13222, Amsterdam'],
        ['Franz', '03.05.2020', '03.05.2020', '11111, Somewhere', '11221, Somewhere2'],
        ['Franz', '03.05.2020', '05.05.2020', '11223, Upsalla', '14231, Berlin']]

df = pd.DataFrame(data)
df = df.reset_index()
df.columns = ['Delivery_id', 'Driver', 'Date loading', 'Date unloading', 'Loading Address', 'Unloading Address']

df_loading = df[['Delivery_id', 'Driver', 'Date loading', 'Loading Address']]
df_loading['Loading'] = 'Loading'
df_loading.columns = ['Delivery_id', 'Driver', 'Date', 'Address', 'Loading']
df_unloading = df[['Delivery_id', 'Driver', 'Date unloading', 'Unloading Address']]
df_unloading['Loading'] = 'Unloading'
df_unloading.columns = ['Delivery_id', 'Driver', 'Date', 'Address', 'Loading']
df = pd.concat([df_loading, df_unloading])

Next, converting the date column from string into datetime so that Pandas understands it as a date.

df['Date'] = pd.to_datetime(df['Date'], format='%d.%m.%Y')

Then it's as easy as setting the index to the values we want to sort by, and sorting it:

df = df.set_index(['Driver', 'Date', 'Delivery_id', 'Loading']).sort_index()

Output:

print(df)

                                                   Address
Driver Date       Delivery_id Loading                     
Franz  2020-05-03 2           Loading     11111, Somewhere
                              Unloading  11221, Somewhere2
                  3           Loading       11223, Upsalla
       2020-05-05 3           Unloading      14231, Berlin
Peter  2020-05-02 0           Loading        12342, Berlin
       2020-05-03 0           Unloading     14221, Utrecht
                  1           Loading       14221, Utrecht
       2020-05-04 1           Unloading   13222, Amsterdam

Transposed output, if you prefer the horizontal format:

print(df.T.to_string())


Driver                  Franz                                                            Peter                                                  
Date               2020-05-03                                        2020-05-05     2020-05-02      2020-05-03                        2020-05-04
Delivery_id                 2                                  3              3              0               0               1                 1
Loading               Loading          Unloading         Loading      Unloading        Loading       Unloading         Loading         Unloading
Address      11111, Somewhere  11221, Somewhere2  11223, Upsalla  14231, Berlin  12342, Berlin  14221, Utrecht  14221, Utrecht  13222, Amsterdam

And if you wanted to keep the vertical sorting by driver and have the rest of the data horizontal, then you could do:

idx = pd.IndexSlice
for driver in df.T.columns.get_level_values(0).unique():
    print(df.loc[idx[driver, :, :]].T.to_string())
    print()

Driver                  Franz                                                  
Date               2020-05-03                                        2020-05-05
Delivery_id                 2                                  3              3
Loading               Loading          Unloading         Loading      Unloading
Address      11111, Somewhere  11221, Somewhere2  11223, Upsalla  14231, Berlin

Driver               Peter                                                  
Date            2020-05-02      2020-05-03                        2020-05-04
Delivery_id              0               0               1                 1
Loading            Loading       Unloading         Loading         Unloading
Address      12342, Berlin  14221, Utrecht  14221, Utrecht  13222, Amsterdam
Related