Re-shaping pandas dataframe to match specific output

user_6415080 0
Viewed 93

I am struggling to find some elegant solution to get what I need from my data. I am able to get what I want but with too much efforts, which I believe can be done quite better and that's what I am looking for.

So here is sample of my DataFrame

>>> df = pd.DataFrame({'device_name': ['tap_switch_1', 'tap_switch_1', 'tap_switch_1', 'tap_switch_1', 'tap_switch_1', 'tap_switch_1', 'tap_switch_1', 'tap_switch_1', 'tap_switch_1'], 'interface': ['ethernet3', 'ethernet4', 'ethernet38', 'ethernet7', 'ethernet8', 'ethernet31', 'ethernet1', 'ethernet12', 'ethernet20'], 'tap_port': ['1-tx-a-rx', '1-tx-b-rx', '1-b', '2-tx-a-rx', '2-tx-b-rx', '2-b', '3-tx-a-rx', '3-tx-b-rx', '3-b'], 'switch_name': ['sw_ag1', 'sw_ag1', 'sw_client1', 'sw_ag1', 'sw_ag1', 'sw_client2', 'sw_ag1', 'sw_ag1', 'sw_client3']})

Current DataFrame shape:

device_name     interface   tap_port    switch_name
tap_switch_1    ethernet3   1-tx-a-rx   sw_ag1
tap_switch_1    ethernet4   1-tx-b-rx   sw_ag1
tap_switch_1    ethernet38  1-b         sw_client1
tap_switch_1    ethernet7   2-tx-a-rx   sw_ag1
tap_switch_1    ethernet8   2-tx-b-rx   sw_ag1
tap_switch_1    ethernet31  2-b         sw_client2
tap_switch_1    ethernet1   3-tx-a-rx   sw_ag1
tap_switch_1    ethernet12  3-tx-b-rx   sw_ag1
tap_switch_1    ethernet20  3-b         sw_client3

The desired output I am looking to get is this:

device_name   id  agg_switch   rx_int     tx_int    client_switch client_port
tap_switch_1   1  sw_ag1       ethernet3  ethernt4  sw_client1    ethernet38
tap_switch_1   2  sw_ag1       ethernet7  ethernt8  sw_client2    ethernet31
tap_switch_1   3  sw_ag1       ethernet1  ethernt12 sw_client3    ethernet20

The logic

So basically I have network setup where one switch is used to tap multiple interfaces. For every client switch port there are two device_name interfaces - one for each direction (RX/TX). I am using tap_port name to combine all that into one group, based on first integer I see, which indicates the "tap_group".

Current "solution"

Here is the way I am doing it now, which I don't quite and it doesn't give me the desired output:

# Add new `id` column
>>> df['id']=df.tap_port.str[0]

# Get RX/TX direction as new column `direction`
>>> df['direction']=df.tap_port.apply(lambda x: x[-4:] if 'x' in x else '-')

# Trying to get the desired output
>>> df.pivot(index='id', columns='direction')[['switch_name','interface']]
          switch_name                   interface
direction           -    a-rx    b-rx           -       a-rx        b-rx
id
1          sw_client1  sw_ag1  sw_ag1  ethernet38  ethernet3   ethernet4
2          sw_client2  sw_ag1  sw_ag1  ethernet31  ethernet7   ethernet8
3          sw_client3  sw_ag1  sw_ag1  ethernet20  ethernet1  ethernet12

This is very close to what I need, but not quite as per the desired output.

Many thanks in advance for your help!

pandas python
2 Answers

I dont think this is an optimal solution , but it is the desired output.

df = pd.DataFrame({'device_name': ['tap_switch_1', 'tap_switch_1', 'tap_switch_1', 'tap_switch_1', 'tap_switch_1', 'tap_switch_1', 'tap_switch_1', 'tap_switch_1', 'tap_switch_1'], 'interface': ['ethernet3', 'ethernet4', 'ethernet38', 'ethernet7', 'ethernet8', 'ethernet31', 'ethernet1', 'ethernet12', 'ethernet20'], 'tap_port': ['1-tx-a-rx', '1-tx-b-rx', '1-b', '2-tx-a-rx', '2-tx-b-rx', '2-b', '3-tx-a-rx', '3-tx-b-rx', '3-b'], 'switch_name': ['sw_ag1', 'sw_ag1', 'sw_client1', 'sw_ag1', 'sw_ag1', 'sw_client2', 'sw_ag1', 'sw_ag1', 'sw_client3']})
df['col']=pd.DataFrame([i.split('-') for i in df['tap_port']])[2].fillna('').replace({'a':'rx_int','b':'tx_int','':'client port'})
df['id']=[i.split('-')[0] for i in df['tap_port']]
df_pvt=df.pivot(index='id',columns='col',values='interface').reset_index()
x=df_pvt.join(df[['switch_name','device_name']][df['col']=='client port'].rename(columns={'switch_name':'client switch'}).reset_index(drop=True))
final=x.join(df[['switch_name']][df['col']=='tx_int'].rename(columns={'switch_name':'agg_switch'}).reset_index(drop=True))

Out[126]: 
  id client port     rx_int      tx_int client switch   device_name agg_switch
0  1  ethernet38  ethernet3   ethernet4    sw_client1  tap_switch_1     sw_ag1
1  2  ethernet31  ethernet7   ethernet8    sw_client2  tap_switch_1     sw_ag1
2  3  ethernet20  ethernet1  ethernet12    sw_client3  tap_switch_1     sw_ag1
user_109535920

Okay, just for the sake of having this in the SO history, here is the solution I came-up with. Half-way through I realised I may have two different agg_switch variables so I had to introduce a separate column (agg_switch2)

cols=['device_name','id','client_port','rx_int','tx_int','client_switch','agg_switch', 'agg_switch2']

>>> new_df=(df
...    .set_index(['device_name','id','direction'])
...    .unstack('direction')[['interface','switch_name']]
...    .reset_index(col_level=-2)
...    .droplevel(0, axis=1)
)

>>> new_df.columns = cols

>>> new_df

    device_name id client_port     rx_int      tx_int client_switch agg_switch agg_switch2
0  tap_switch_1  1  ethernet38  ethernet3   ethernet4    sw_client1     sw_ag1      sw_ag1
1  tap_switch_1  2  ethernet31  ethernet7   ethernet8    sw_client2     sw_ag1      sw_ag1
2  tap_switch_1  3  ethernet20  ethernet1  ethernet12    sw_client3     sw_ag1      sw_ag1
user_64150800
Related