Corroborating 2 large Python dictionaries

user_6017833 0
Viewed 75

Say I have a 2 dictionaries, each with around 100000 entries (each can be of different length):

dict1 = {"a": ["w", "x"], "b":["y"], "c":["z"] ...}
dict2 = {"x": ["a", "b"], "y":["b", "d"], "z":["d"] ...}

I need to perform an operation using these two dictionaries:

  • Treat each dict item as a set of mapping (i.e list of all mappings in dict1 would be"a"->"w", "a"->"x", "b"->"y" and "c"->"z")
  • Only keep mappings in dict1 if the reverse mapping exists in dict2.

The resulting dictionary would be: {"a": ["x"], "b", ["y"]}

My current solution uses 2 m*n all zeros dataframes where m and n are the lengths of dict1 and dict2 respectively and the index labels are the keys in dict1 and the column labels are the keys in dict2.

For the first dataframe, I insert a 1 at each value where the index label -> column label represent a mapping in dict1. For the second dataframe, I insert a 1 at each value where the column label -> index label represent a mapping in dict2.

I then perform an element-size product between the two dataframes, which only leaves values that have a mapping "a1"->"x1" in dict1 and "x1"->"a1" in dict2.

However, this takes up way too much memory and is very expensive. Is there an alternative algorithm I can use?

pandas dataframe dictionary python
2 Answers

How about to use the same idea, but replace a sparse matrix you're using with a set of key pairs? Something like:

import collections
def fn(dict1, dict2):
    mapping_set = set()
    for k, vv in dict2.items():
        for v in vv:
            mapping_set.add((k, v))
    result_dict = collections.defaultdict(list)
    for k, vv in dict1.items():
        for v in vv:
            if (v, k) in mapping_set:  # Note reverse order of k and v
                result_dict[k].append(v)
    return result_dict

Update: It will use O(total number of values in dict2) of memory and O(total number of values in dict1) + O(total number of values in dict2) time - both a linear. It's not possible to solve the problem algorithmically faster as every value in every dict has to be visited at least once.

user_2347450

Given that you have python objects to begin with, you may want to stay in the python domain. If you need to iterate through the entire dict to create your matrix anyway, you may find that filtering in-place doesn't take much longer.

default = ()
result = {k: v for k, v in dict1.items() if k in dict2.get(v, default)}

If your list are short, this will be totally fine. If they contain many elements, linear search will start to compare to the overhead of set lookup. In that case, you may want to preprocess dict2 to contain sets rather than lists:

dict2 = {k: set(v) for k, v in dict2.items}

or in-place

for k, v in dict2.items():
    dict2[k] = set(v)
user_29887300
Related