Finding identical key/value pairs in list of dict without the use of nested loop

user_12179137 0
Viewed 276

I am doing a very simple computation, where I find identical key/value pairs in a list of dicts to combine them by summing them up. Lets say the data is:

Edit: name & id are arbitrary names for example, I have a very large dict where I use multiple keys

Input

{
  "name":"first",
  "id":"1234",
  "quantity":10
},
{
  "name":"first",
  "id":"1234",
  "quantity":30
},
{
  "name":"another",
  "id":"0000",
  "quantity":10
}

Output

{
 "name":"first",
  "id":"1234",
  "quantity":40
},
{
  "name":"another",
  "id":"0000",
  "quantity":10
}

I am curious to learn on how to do this the "pythonic" way, avoiding nested loops if possible.

For now I have it as such which I am not happy with:

for entry in quantities:
    for compare in quantities:
        if id(entry) != id(compare):
            if (entry["name"] == compare["name"]) and (entry["id"] == compare["id"]):
                entry["quantity"] = entry["quantity"] + compare["quantity"]
                quantities.remove(compare)

Any tips/suggestions are appreciated, thanks!

dictionary python
2 Answers

Use another dictionary and group on your key, by that I mean "name" and "id" (although, will "id" not be sufficient? Misleading if it won't).

Something like:

grouper = {}
for q in quantities:
    key = q['name'], q['id']
    if key in grouper:
        grouper[key]['quantity'] += q['quantity']
    else:
        grouper[key] = q.copy()
quantities = list(grouper.values())

In a REPL:

In [1]: quantities = [
   ...: {
   ...:   "name":"first",
   ...:   "id":"1234",
   ...:   "quantity":10
   ...: },
   ...: {
   ...:   "name":"first",
   ...:   "id":"1234",
   ...:   "quantity":30
   ...: },
   ...: {
   ...:   "name":"another",
   ...:   "id":"0000",
   ...:   "quantity":10
   ...: }
   ...: ]

In [2]: grouper = {}

In [3]: for q in quantities:
   ...:     key = q['name'], q['id']
   ...:     if key in grouper:
   ...:         grouper[key]['quantity'] += q['quantity']
   ...:     else:
   ...:         grouper[key] = q.copy()
   ...:

In [4]: grouper
Out[4]:
{('first', '1234'): {'name': 'first', 'id': '1234', 'quantity': 40},
 ('another', '0000'): {'name': 'another', 'id': '0000', 'quantity': 10}}

Then you can get your new list from the values directly:

In [5]: list(grouper.values())
Out[5]:
[{'name': 'first', 'id': '1234', 'quantity': 40},
 {'name': 'another', 'id': '0000', 'quantity': 10}]

This approach will take linear time and linear space.

Note, q.copy() creates a shallow copy, which is fine here, but may not be if you had had mutable values in your dict.

Also note, you may want to just reconsider your data structure. Do you really want a list? Probably, you want some type of dict if you have a unique key and want to be able to find the object by that key quickly.

user_50144550

Mehod 1: Using groupby and reduce

from itertools import groupby
from functools import reduce

def merge(d1, d2):
    ' merge two dictionaries based upon summing key values not in grouper '
    return {k:v if k in grouper else v + d2.get(k, 0) for k, v in d1.items()}

grouper = ("name", "id")  # keys to groupby
lst.sort(key = lambda d:[d[key] for key in grouper])  # Sort list inplace based upon grouper keys
                                                      # Done inplace to save space
# Merge dicts in list in same group based upon merge function
outputlist = [(reduce(merge, g)) for _, g in groupby(lst, lambda d:[d[key] for key in grouper])]
    

[{'name': 'another', 'id': '0000', 'quantity': 10},
 {'name': 'first', 'id': '1234', 'quantity': 40}]

Method 2--Using Pandas

A one-liner that avoids all loops (method essentially replicates Method 1)

outputlist = pd.DataFrame(lst).groupby(['name', 'id']).sum().reset_index().to_dict('records')

outputlist:

[{'name': 'another', 'id': '0000', 'quantity': 10},
     {'name': 'first', 'id': '1234', 'quantity': 40}]

Explanation

pd.DataFrame(lst)            - generate pandas DataFrame from list of dictionaries
groupby(['name', 'id'])      - group rows by name & id
sum()                        - sum the non-grouped values in each group
reset_index()                - reset index back to 0, 1, 2, ...
to_dict('records')           - convert to list of dictionaries 
                               with each row data as dictionary
user_30660770
Related