How to sort dataframe without changing groupings in Pandas?

Viewed 389

I am trying to do a groupby using Pandas and apply a sort. Something like below: 1 I have so far created the individual frames to get the subtotals. Not sure how to proceed after that to get the sorting done properly without resorting to hacks.

Sample dataframe is from this previous question.

df = pd.DataFrame({
'admin0': ['cntry1', 'cntry1', 'cntry1', 'cntry1', 'cntry1', 'cntry1', 'cntry2', 'cntry2', 'cntry2', 'cntry2', 'cntry2'], 
'admin1': ['state1', 'state1', 'state1', 'state2', 'state2', 'state2', 'state3', 'state3', 'state3', 'state3', 'state4'], 
'admin2': ['city1', 'city1', 'city2', 'city3', 'city4', 'city4', 'city5', 'city6', 'city6', 'city6', 'city7'], 
'windspeed': [60, 90, 60, 60, 60, 90, 60, 60, 90, 120, 60], 
'population': [700, 210, 100, 70, 180, 370, 890, 120, 420, 360, 740]
})
g1 = df.groupby(['admin0', 'admin1',  'admin2']).sum()
g2 = g1.groupby(level=[0, 1]).sum()
g2.index = pd.MultiIndex.from_arrays([g2.index.get_level_values(0), g2.index.get_level_values(1), len(g2.index)*['']])
g3 = g1.groupby(level=0).sum()
g3.index = pd.MultiIndex.from_arrays([g3.index.get_level_values(0), len(g3.index)*[''], len(g3.index)*['']])
g = pd.concat([g1, g2, g3])

State of g now:

                         windspeed  population
admin0 admin1 admin2                       
cntry1 state1 city1         150         910
              city2          60         100
       state2 city3          60          70
              city4         150         550
cntry2 state3 city5          60         890
              city6         270         900
       state4 city7          60         740
cntry1 state1               210        1010
       state2               210         620
cntry2 state3               330        1790
       state4                60         740
cntry1                      420        1630
cntry2                      390        2530

I now want to be able to sort without changing the grouping as shown in the gif.

Expected response when grouping by windspeed ascending:

                      windspeed  population
admin0 admin1 admin2
cntry2                      390        2530
       state4                60         740
              city7          60         740
       state3               330        1790
              city5          60         890
              city6         270         900
cntry1                      420        1630
       state1               210        1010
              city2          60         100
              city1         150         910
       state2               210         620
              city3          60          70
              city4         150         550
5 Answers

Example for ascending sort by 'windspeed', continuing from your dataframe g:

levels = ['admin0', 'admin1',  'admin2']
g.groupby(levels[:-1], group_keys = False).apply(lambda x: x.sort_values(by = 'windspeed', ascending=True))

Basically you need to perform groupby as you did, then groupby it again on all levels except the last one and apply a sort by a desired column.

edit: after clarification on question, revising answer, original response below hline

To sort values within a multi-index, just include the multi-index levels in the sort call. Sorting happens in order, so it sorts by the first key, then within that sorts the second key, and so on.:

>>> g.sort_values(['admin0', 'admin1', 'admin2', 'windspeed'])
                      windspeed  population
admin0 admin1 admin2                       
cntry1                      420        1630
       state1               210        1010
              city1         150         910
              city2          60         100
       state2               210         620
              city3          60          70
              city4         150         550
cntry2                      390        2530
       state3               330        1790
              city5          60         890
              city6         270         900
       state4                60         740
              city7          60         740

However in your example the ascending=False looks like this, which isn't exactly what you want, because of the way the multi-index was created with '' blank strings for the 'country' and 'state'-level measurements:

>>> g.sort_values(['admin0', 'admin1', 'admin2', 'windspeed'], ascending=False)
                      windspeed  population
admin0 admin1 admin2                       
cntry2 state4 city7          60         740
                             60         740
       state3 city6         270         900
              city5          60         890
                            330        1790
                            390        2530
cntry1 state2 city4         150         550
              city3          60          70
                            210         620
       state1 city2          60         100
              city1         150         910
                            210        1010
                            420        1630

this might be sorta confusing, but it's because of the way pandas prints multiindexes -- when a level of a multi-index is repeated, it isn't printed again. but since the level values are set to '' you can't tell when it's just not repeating them again vs. when the label is actually '' You can get the exact output you want by using some character that comes after state or city , like for example if i replace '' with 'z' in the MultiIndex.from_arrays calls:

g.sort_values(['admin0', 'admin1', 'admin2', 'windspeed'], ascending=False)
                      windspeed  population
admin0 admin1 admin2                       
cntry2 z      z             390        2530
       state4 z              60         740
              city7          60         740
       state3 z             330        1790
              city6         270         900
              city5          60         890
cntry1 z      z             420        1630
       state2 z             210         620
              city4         150         550
              city3          60          70
       state1 z             210        1010
              city2          60         100
              city1         150         910

Or you could use an ordered categorical index where you explicitly specify that '' should be the last (rather than the first) label.


original answer

I think I may be misunderstanding your question, is this not what you want?

>>> g.sort_values(by="windspeed")
                      windspeed  population
admin0 admin1 admin2                       
cntry1 state1 city2          60         100
       state2 city3          60          70
cntry2 state3 city5          60         890
       state4 city7          60         740
                             60         740
cntry1 state1 city1         150         910
       state2 city4         150         550
       state1               210        1010
       state2               210         620
cntry2 state3 city6         270         900
                            330        1790
                            390        2530
cntry1                      420        1630

>>> g.sort_values(by=["windspeed","population"])
                      windspeed  population
admin0 admin1 admin2                       
cntry1 state2 city3          60          70
       state1 city2          60         100
cntry2 state4 city7          60         740
                             60         740
       state3 city5          60         890
cntry1 state2 city4         150         550
       state1 city1         150         910
       state2               210         620
       state1               210        1010
cntry2 state3 city6         270         900
                            330        1790
                            390        2530
cntry1                      420        1630
>>> 

Use sort_index(level=0)

                      windspeed  population
admin0 admin1 admin2                       
cntry1                      420        1630
       state1               210        1010
              city1         150         910
              city2          60         100
       state2               210         620
              city3          60          70
              city4         150         550
cntry2                      390        2530
       state3               330        1790
              city5          60         890
              city6         270         900
       state4                60         740
              city7          60         740

You need to include the first 3 columns in the sort

g1.sort_values(by=['admin0', 'admin1',  'admin2','windspeed','population'], ascending = False)

This is my hacked solution. It is ugly and definitely bad due to excessive computation in python layer. Hope pandas/numpy/scipy experts can poke holes and give a better solution than this.

import pandas as pd
import numpy as np

def compute_glevels(mini_dfs, rp_len):

    level_vals = [{} for _ in range(rp_len)]
    for i, sdf in enumerate(reversed(mini_dfs)):
        idx = sdf.index
        g_levels = [[[] for _ in range(rp_len)] for _ in range(len(idx))]
        for j, entry in enumerate(sdf.index):
            for l, lv in enumerate(entry):
                if lv:
                    if l > i:
                        print('i', i,  'j', j, 'l', l, 'level_vals', level_vals)
                        raise Exception
                    else:
                        v = level_vals[l].setdefault(entry[:l+1], len(level_vals[l]) + 1)
                        g_levels[j][l] = v
                else:
                    g_levels[j][l] = 0
        sdf['g_lvl'] = g_levels


def run(df, row_pivots=None, aggregates=None, sort=None):

    rp_len = len(row_pivots)
    by = list(row_pivots[:-1])
    asc = [True] * (rp_len - 1)
    for entry in sort:
        by.append(entry[0])
        asc.append(entry[1] == 'asc')
    grouper = df.groupby(list(row_pivots), sort=False).agg(aggregates)
    grouper.sort_values(by=by, ascending=asc, inplace=True)
    sdfs = [grouper]

    for i in range(1, rp_len):
        if i < rp_len - 1:
            by = list(row_pivots[:-(i+1)])
            asc = [True] * len(by)
        else:
            by = []
            asc = []
        for entry in sort:
            by.append(entry[0])
            asc.append(entry[1] == 'asc')
        levels = list(range(rp_len - i))
        sdf = sdfs[i-1].groupby(level=levels, sort=False).agg(aggregates)
        sdf.sort_values(by=by, ascending=asc, inplace=True)
        idx_len = len(sdf.index)
        sdf.index = pd.MultiIndex.from_arrays([sdf.index.get_level_values(l) if l < (rp_len - i) else [''] * idx_len for l in range(rp_len)])
        sdfs.append(sdf)

    compute_glevels(sdfs, rp_len)
    total_dict = sdfs[-1].agg(aggregates).to_dict()
    total_dict['g_lvl'] = [0, 0, 0]
    total = pd.DataFrame([total_dict], columns=sdfs[0].columns, index=pd.MultiIndex.from_tuples([[''] * rp_len], names=row_pivots))
    sdfs.append(total)
    g = pd.concat(sdfs).sort_values('g_lvl')
    print(g)
    return g

if __name__ == '__main__':

    df = pd.DataFrame({
        'admin0': ['cntry1', 'cntry1', 'cntry1', 'cntry1', 'cntry1', 'cntry1', 'cntry2', 'cntry2', 'cntry2', 'cntry2','cntry2'],
        'admin1': ['state1', 'state1', 'state1', 'state2', 'state2', 'state2', 'state3', 'state3', 'state3', 'state3','state4'],
        'admin2': ['city1', 'city1', 'city2', 'city3', 'city4', 'city4', 'city5', 'city6', 'city6', 'city6', 'city7'],
        'windspeed': [60, 90, 60, 60, 60, 90, 60, 60, 90, 120, 60],
        'population': [700, 210, 100, 70, 180, 370, 890, 120, 420, 360, 740]
    })
    g1 = run(df, row_pivots=('admin0', 'admin1', 'admin2'), aggregates={'windspeed': np.sum, 'population': np.sum}, sort=(('windspeed', 'asc'), ))
    g2 = run(df, row_pivots=('admin0', 'admin1', 'admin2'), aggregates={'windspeed': np.sum, 'population': np.sum}, sort=(('windspeed', 'asc'), ('population', 'asc')))
    print(g1.equals(g2))
Related