Python group by

Viewed 216741

Assume that I have a set of data pair where index 0 is the value and index 1 is the type:

input = [
          ('11013331', 'KAT'), 
          ('9085267',  'NOT'), 
          ('5238761',  'ETH'), 
          ('5349618',  'ETH'), 
          ('11788544', 'NOT'), 
          ('962142',   'ETH'), 
          ('7795297',  'ETH'), 
          ('7341464',  'ETH'), 
          ('9843236',  'KAT'), 
          ('5594916',  'ETH'), 
          ('1550003',  'ETH')
        ]

I want to group them by their type (by the 1st indexed string) as such:

result = [ 
           { 
             'type': 'KAT', 
             'items': ['11013331', '9843236'] 
           },
           {
             'type': 'NOT', 
             'items': ['9085267', '11788544'] 
           },
           {
             'type': 'ETH', 
             'items': ['5238761', '962142', '7795297', '7341464', '5594916', '1550003'] 
           }
         ] 

How can I achieve this in an efficient way?

9 Answers

This answer is similar to @PaulMcG's answer but doesn't require sorting the input.

For those into functional programming, groupBy can be written in one line (not including imports!), and unlike itertools.groupby it doesn't require the input to be sorted:

from functools import reduce # import needed for python3; builtin in python2
from collections import defaultdict

def groupBy(key, seq):
 return reduce(lambda grp, val: grp[key(val)].append(val) or grp, seq, defaultdict(list))

(The reason for ... or grp in the lambda is that for this reduce() to work, the lambda needs to return its first argument; because list.append() always returns None the or will always return grp. I.e. it's a hack to get around python's restriction that a lambda can only evaluate a single expression.)

This returns a dict whose keys are found by evaluating the given function and whose values are a list of the original items in the original order. For the OP's example, calling this as groupBy(lambda pair: pair[1], input) will return this dict:

{'KAT': [('11013331', 'KAT'), ('9843236', 'KAT')],
 'NOT': [('9085267', 'NOT'), ('11788544', 'NOT')],
 'ETH': [('5238761', 'ETH'), ('5349618', 'ETH'), ('962142', 'ETH'), ('7795297', 'ETH'), ('7341464', 'ETH'), ('5594916', 'ETH'), ('1550003', 'ETH')]}

And as per @PaulMcG's answer the OP's requested format can be found by wrapping that in a list comprehension. So this will do it:

result = {key: [pair[0] for pair in values],
          for key, values in groupBy(lambda pair: pair[1], input).items()}
result = []
# Make a set of your "types":
input_set = set([tpl[1] for tpl in input])
>>> set(['ETH', 'KAT', 'NOT'])
# Iterate over the input_set
for type_ in input_set:
    # a dict to gather things:
    D = {}
    # filter all tuples from your input with the same type as type_
    tuples = filter(lambda tpl: tpl[1] == type_, input)
    # write them in the D:
    D["type"] = type_
    D["itmes"] = [tpl[0] for tpl in tuples]
    # append D to results:
    result.append(D)

result
>>> [{'itmes': ['9085267', '11788544'], 'type': 'NOT'}, {'itmes': ['5238761', '5349618', '962142', '7795297', '7341464', '5594916', '1550003'], 'type': 'ETH'}, {'itmes': ['11013331', '9843236'], 'type': 'KAT'}]

You could use convtools library which generates ad-hoc code for your exact task and allows for dynamic code generation.

from convtools import conversion as c

# grouping by second elements of tuples;
# aggregate defines the schema of the expected output elements
converter = c.group_by(c.item(1)).aggregate({
    "type": c.item(1),
    "items": c.ReduceFuncs.Array(c.item(0)),
}).gen_converter()

# now you have a function which does what you asked,
# store it somewhere for further reuse
converter(input_data)

Following Snippet is also a way to get the desired results -

res = []
dict1 = {}
for item in input:
  if item[1] not in dict1:
    dict1[item[1]] = [item[0]]
  elif item[1] in dict1:
    dict1[item[1]].append(item[0])
for k, v in dict1.items():
  res.append({"type": k, "items": v})


# res = [ { type:'KAT', items: ['11013331', '9843236'] },{ type:'NOT',  items: ['9085267', '11788544'] },{ type:'ETH',  items: ['5238761', '962142', '7795297', '7341464', '5594916', '1550003'] }] 

This is not very efficient, but it is Pythonic. Basically, work out the distinct groups by taking the set of group values, and then for each of these groups, get the items that are in that group.

[
    {
        "type": group,
        "items": [item[0] for item in input if item[1] == group]
    }
    for group in {item[1] for item in input}
]
Related