Python3 How to join two lists of dicts by unique keys

user_6241554 0
Viewed 149

I have two lists:

list1 = [ {'sth': 13, 'important_key1': 'AA', 'important_key2': '3'}, {'oh!': 14, 'important_key1': 'FF', 'important_key2': '4'}, {'sth_else': 'abc', 'important_key1': 'ZZ', 'important_key2': '5'}]
list2 = [ {'why-not': 'tAk', 'important_key1': 'GG', 'important_key2': '4'}, {'hmmm': 'no', 'important_key1': 'AA', 'important_key2': '3'}]

I want to return a list with objects only from list1 but if the same important_key1 and important_key2 is in any element in list2 I want this element from list2.

So the output should be:

[ {'hmmm': 'no', 'important_key1': 'AA', 'important_key2': '3'}, {'oh!': 14, 'important_key1': 'FF', 'important_key2': '4'}, {'sth_else': 'abc', 'important_key1': 'ZZ', 'important_key2': '5'}]

It is not complicated to do it by two or three loops but I wonder whether there is a simple way by using list comprehensions or something like that.

This is the "normal" way:

list1 = [ {'sth': 13, 'important_key1': 'AA', 'important_key2': '3'}, {'oh!': 14, 'important_key1': 'FF', 'important_key2': '4'}]
list2 = [ {'hmmm': 'no', 'important_key1': 'AA', 'important_key2': '3'}, {'why-not': 'tAk', 'important_key1': 'GG', 'important_key2': '4'}]

final_list = []
for element in list1:
    there_was_in_list2 = False
    for another_element in list2:
        if element['important_key1'] == another_element['important_key1'] and element['important_key2'] == another_element['important_key2']:
            final_list.append(another_element)
            there_was_in_list2 = True
            break
    if not there_was_in_list2:
        final_list.append(element)
print(final_list)

is there any Pythonic way to do that?

list-comprehension python
5 Answers

You can use a list comprehension:

list1 = [{'sth': 13, 'important_key1': 'AA', 'important_key2': '3'}, {'oh!': 14, 'important_key1': 'FF', 'important_key2': '4'}]
list2 = [{'hmmm': 'no', 'important_key1': 'AA', 'important_key2': '3'}, {'why-not': 'tAk', 'important_key1': 'GG', 'important_key2': '4'}]
vals = ['important_key1', 'important_key2']
new_list = [[c if any(c[a] == i[a] for a in vals) else i for c in list2] for i in list1]
final_result = [i[0] for i in new_list if i]

Output:

[{'hmmm': 'no', 'important_key1': 'AA', 'important_key2': '3'}, {'oh!': 14, 'important_key1': 'FF', 'important_key2': '4'}]
user_73267380

You can convert list2 to a dict indexed by a tuple of the values the important keys in list2, and then use it to determine if the same keys in list1 have the same values as you iterate through list1 in a list comprehension, so that the time complexity gets reduced to O(n) from your O(n*m):

keys = ['important_key1', 'important_key2']
d2 = {tuple(d[k] for k in keys): d for d in list2[::-1]}
print([d2.get(tuple(d[k] for k in keys), d) for d in list1])

This outputs (with your sample input):

[{'hmmm': 'no', 'important_key1': 'AA', 'important_key2': '3'}, {'oh!': 14, 'important_key1': 'FF', 'important_key2': '4'}, {'sth_else': 'abc', 'important_key1': 'ZZ', 'important_key2': '5'}]

As you described in your question, only {'sth': 13, 'important_key1': 'AA', 'important_key2': '3'} in list1 would get replaced by {'hmmm': 'no', 'important_key1': 'AA', 'important_key2': '3'} because only this dict has both important_key1 and important_key2 matching those of a dict in list2.

user_68909120

You can spare the there_was_in_list2 variable by using for...else. The else statement will be executed when the previous for loop finished normally (i.e. it was not "broken").

final_list = []
for element in list1:
    for another_element in list2:
        if element['important_key1'] == another_element['important_key1'] and element['important_key2'] == another_element['important_key2']:
            final_list.append(another_element)
            break
    else:
        final_list.append(element)
user_37672390

If you want your code to be more concise yet maintain readability you can replace the second for loop with a combined next and filter:

final_list.append(next(
    filter(lambda x: ..., list2),
    element  # Default in case filter yields nothing.
))
user_37672390

Most all other paths are covered so here is another idea, just coming up with as many possible routes we can, this was fun btw thank you :)

l3 = l1[:]

for idx, item in enumerate(l2):    
    for x, i in enumerate(l1):
        k = list(zip(item.values(), i.values())) 
        if len(set(k[1])) < len(k[1]) and len(set(k[2])) < len(k[2]):
            l3[x] = item

print(l3)

(xenial)vash@localhost:~/python/stack_overflow/sept$ python3.7 uniq.py
[{'hmmm': 'no', 'important_key1': 'AA', 'important_key2': '3'}, {'oh!':14, 'important_key1': 'FF', 'important_key2': '4'},
{'sth_else': 'abc', 'important_key1': 'ZZ', 'important_key2': '5'}]
user_102556520
Related