Fastest way to filter for certain keys from dictionaries which are stored in a list of dictionaries which is in turn stored in a dictionary

python444444 0
Viewed 45

I am looking for the fastest way to filter for certain keys from "inner" dictionaries inside lists inside an "outer" dictionary, so kind of a tricky data structure.

The data structure (input) looks like this: All inner dictionaries have the same keys but the lists can have different numbers of dictionaries.

d = {1:[{1:33 ,2:33, 3:33, 4:33}, {1:33 ,2:33, 3:33, 4:33}, {1:33 ,2:33, 3:33, 4:33}],
 2:[{1:33 ,2:33, 3:33, 4:33}],
 3:[{1:33 ,2:33, 3:33, 4:33}, {1:33 ,2:33, 3:33, 4:33}]}

What I want as output is the same structure but only the inner dictionary keys 1 and 2, I don't need 3 and 4 anymore:

d_new = {1: [{1: 33, 2: 33}, {1: 33, 2: 33}, {1: 33, 2: 33}],
 2: [{1: 33, 2: 33}],
 3: [{1: 33, 2: 33}, {1: 33, 2: 33}, {1: 33, 2: 33}]}

So far, I have two approaches which solve the problem but are very slow:

  1. Combination of list- and dict comprehension:
d_new = {n: [{1: d[1], 2: d[2]} for d in d[n]] for n in d.keys()}
  1. For-Loop
d_new = {}

keys = d.keys()

for key in keys:
    storage = []
    for x in d[key]:
        storage.append({1: x[1], 2: x[2]})
    d_new[key] = storage

They both solve the problem but are very slow when applied to large dictionaries so I am looking for a faster and more efficient way, maybe it is also possible to apply some kind of paralellization here. Very happy for any help!

filtering dictionary list python-3.x python
1 Answers

The Code

from copy import deepcopy

d = {1:[{1:33 ,2:33, 3:33, 4:33}, {1:33 ,2:33, 3:33, 4:33}, {1:33 ,2:33, 3:33, 4:33}],
 2:[{1:33 ,2:33, 3:33, 4:33}],
 3:[{1:33 ,2:33, 3:33, 4:33}, {1:33 ,2:33, 3:33, 4:33}]}

d_new = deepcopy(d) #Done so that we don't run into issues with list items

d_items = list(d_new.items())
for i in d_items:
    for j in i[1]:
        pop_l = [] #Created a list of entries to pop since popping them while in the for loop changes the loop length and gives a runtime error
        for k in j:
            if k > 2:
                pop_l.append(k)
        for x in pop_l: #Popping all unwanted keys
            j.pop(x)

print(d_new)

Output

{1: [{1: 33, 2: 33}, {1: 33, 2: 33}, {1: 33, 2: 33}], 2: [{1: 33, 2: 33}], 3: [{1: 33, 2: 33}, {1: 33, 2: 33}]}
EliteGamerSiddhu0
Related