Can I reduce a tuple list by key using Python?

Viewed 120

I am currently working on showing some visuals about how my NER model has performed. The data I currently have looks like this:

counter_list = [
    ('Name', {'p':0.56,'r':0.56,'f':0.56}),
    ('Designation', {'p':0.10,'r':0.20,'f':0.14}),
    ('Location', {'p':0.56,'r':0.56,'f':0.56}),
    ('Name', {'p':0.14,'r':0.14,'f':0.14}),
    ('Designation', {'p':0.10,'r':0.20,'f':0.14}),
    ('Location', {'p':0.56,'r':0.56,'f':0.56})
]

I would like to eliminate the duplicates and add their respective values to only one of each kind. So the output to look like this:

[
    ('Name', {'p':0.7,'r':0.7,'f':0.7}),
    ('Designation', {'p':0.2,'r':0.4,'f':0.28}),
    ('Location', {'p':1.12,'r':1.12,'f':1.12})
]

I have tried to use the reduce function but it gives me only the output for 'Name' entry only.

result = functools.reduce(lambda x, y: (x[0], Counter(x[1])+Counter(y[1])) if x[0]==y[0] else (x[0],x[1]), counter_list)

What would be the right approach? I am trying to create some visuals with the final results, to determine which item has the higher 'f','p' or 'r' component.

7 Answers

Why not use pandas and its ~.groupby method?

>>> import pandas as pd
>>> keys, data = zip(*counter_list)
>>> df = pd.DataFrame(data=data, index=keys).groupby(level=0).sum()
>>> df
                p     r     f
Designation  0.20  0.40  0.28
Location     1.12  1.12  1.12
Name         0.70  0.70  0.70

and then do

>>> list(df.T.to_dict().items())
[
    ('Designation', {'p': 0.2, 'r': 0.4, 'f': 0.28}), 
    ('Location', {'p': 1.12, 'r': 1.12, 'f': 1.12}), 
    ('Name', {'p': 0.7, 'r': 0.7, 'f': 0.7})
]

One tricky way might be of using defaultdict and Counter, but it seems a bit unclear about the intentions:

from collections import defaultdict, Counter
result = defaultdict(Counter)
for item, values in counter_list:
    result[item].update(values)
print(result)

This could be done using an intermediate/temporary dictionary. Something like this:

counter_list = [
    ('Name', {'p':0.56,'r':0.56,'f':0.56}),
    ('Designation', {'p':0.10,'r':0.20,'f':0.14}),
    ('Location', {'p':0.56,'r':0.56,'f':0.56}),
    ('Name', {'p':0.14,'r':0.14,'f':0.14}),
    ('Designation', {'p':0.10,'r':0.20,'f':0.14}),
    ('Location', {'p':0.56,'r':0.56,'f':0.56})
]

tdict = dict()

for k, v in counter_list:
    if k not in tdict:
        tdict[k] = v
    else:
        for sk in 'prf':
            tdict[k][sk] += v[sk]

new_list = [(k, v) for k, v in tdict.items()]
print(new_list)

Output:

[('Name', {'p': 0.7000000000000001, 'r': 0.7000000000000001, 'f': 0.7000000000000001}), ('Designation', {'p': 0.2, 'r': 0.4, 'f': 0.28}), ('Location', {'p': 1.12, 'r': 1.12, 'f': 1.12})]

Mine is similar to Lancelot du Lac above. Not the cleanest but it gets your expected output.

counter_list = [
    ('Name', {'p':0.56,'r':0.56,'f':0.56}),
    ('Designation', {'p':0.10,'r':0.20,'f':0.14}),
    ('Location', {'p':0.56,'r':0.56,'f':0.56}),
    ('Name', {'p':0.14,'r':0.14,'f':0.14}),
    ('Designation', {'p':0.10,'r':0.20,'f':0.14}),
    ('Location', {'p':0.56,'r':0.56,'f':0.56})
]

counter_intermediate = {x: {'p': 0, 'r': 0, 'f': 0} for x in list(set([tup[0] for tup in counter_list]))}

for (header, dic) in counter_list:
    for letter, value in dic.items():
        counter_intermediate[header][letter] += value

new_list = [(key, value) for key, value in counter_intermediate.items()]

print(new_list)

Output:

[('Designation', {'p': 0.2, 'r': 0.4, 'f': 0.28}), ('Name', {'p': 0.7000000000000001, 'r': 0.7000000000000001, 'f': 0.7000000000000001}), ('Location', {'p': 1.12, 'r': 1.12, 'f': 1.12})]

A more functional approach could look like the following:

from typing import List, Tuple

def reduce_counter_list(counter_list: List[Tuple]) -> List[Tuple]:
    tmp_dict = {}
    for key, count_dict in counter_list:

        # First time seeing a key, store its value
        if key not in tmp_dict:
            tmp_dict[key] = count_dict

        # Otherwise, add the values to existing vals
        else:
            for k, v in count_dict.items():
                tmp_dict[key][k] += v

    # Finally, return tmp dict as a list of tuples
    return list(tmp_dict.items())

A very compact example where cl = counter_list for space:

l = len(cl)
y = []

for i, x in enumerate(cl[:l//2]):
    y.append((x[0], {k: round(x[1][k]+cl[i+3][1][k], 3) for k in x[1]}))
[
    ('Name', {'f': 0.7, 'p': 0.7, 'r': 0.7}),
    ('Designation', {'f': 0.28, 'p': 0.2, 'r': 0.4}),
    ('Location', {'f': 1.12, 'p': 1.12, 'r': 1.12})
]

This relies on the uniform arrangement of the tuples being in an order of:

[
    ('Name', ...),
    ('Designation', ...),
    ('Location', ...),
    ('Name', ...),
    ('Designation', ...),
    ('Location', ...)
]

You may get name, designation, location (p,r,f) individual lists and sum the values.

counter_list = [
    ('Name', {'p':0.56,'r':0.56,'f':0.56}),
    ('Designation', {'p':0.10,'r':0.20,'f':0.14}),
    ('Location', {'p':0.56,'r':0.56,'f':0.56}),
    ('Name', {'p':0.14,'r':0.14,'f':0.14}),
    ('Designation', {'p':0.10,'r':0.20,'f':0.14}),
    ('Location', {'p':0.56,'r':0.56,'f':0.56})
]

name_list = [(tuplee) for tuplee in counter_list if tuplee[0]=="Name"]
desi_list = [(tuplee) for tuplee in counter_list if tuplee[0]=="Designation"]
loca_list = [(tuplee) for tuplee in counter_list if tuplee[0]=="Location"]

name_p = sum([name[1]['p'] for name in name_list])
name_r = sum([name[1]['r'] for name in name_list])
name_f = sum([name[1]['f'] for name in name_list])

desi_p = sum([desi[1]['p'] for desi in desi_list])
desi_r = sum([desi[1]['r'] for desi in desi_list])
desi_f = sum([desi[1]['f'] for desi in desi_list])

loca_p = sum([loca[1]['p'] for loca in loca_list])
loca_r = sum([loca[1]['r'] for loca in loca_list])
loca_f = sum([loca[1]['f'] for loca in loca_list])

final = [('Name',        {'p': name_p, 'r': name_r, 'f': name_f}), \
         ('Designation', {'p': desi_p, 'r': desi_r, 'f': desi_f}), \
         ('Location',    {'p': loca_p, 'r': loca_r, 'f': loca_f})]
Related