Removing duplicates from list of dicts throws Exception?

user_19445623 0
Viewed 77

I have the following list in python:

[('admin', '', {'type': 'telnet'}), ('admin', '', {'type': 'telnet'})]

Where I'm trying to remove duplicates (something is considered duplicate if it's 100% the same) so I wrote:

my_list = list(dict.fromkeys(my_list))

But sometimes I get the following exception:

unhashable type: 'dict'

How may I fix this?

dictionary python
5 Answers

The issue is that dict.fromkeys uses the elements of the list as the keys of the resultant dictionary. All keys of a dictionary must be hashable. Your list contains tuples. Tuples are hashable if and only if all of their elements are hashable. Dictionaries, which your tuples contain, are not hashable.

For this case, you might be stuck with a crude loop:

unique = []
for elem in my_list:
    if elem not in unique:
        unique.append(elem)
user_80755400

If you want check duplicated and remove them. First, convert each element to str then use set at the end use ast.literal_eval and back them to original like below:

lst = [('admin', '', {'type': 'telnet'}), ('admin', '', {'type': 'telnet'}), 
       ('admn', '', {'type': 'telnet'}), ('admn', '', {'type': 'telnet'}),
# --------^^^-----------------------------^^^ duplicated
       ('admin', '', {'typ': 'telnet'}),('admin', '', {'typ': 'telnet'})]
------------------------^^^----------------------------^^^ duplicated
import ast
list(map(ast.literal_eval, set(map(str,lst))))

output:

[('admin', '', {'type': 'telnet'}),
 ('admn', '', {'type': 'telnet'}),
 ('admin', '', {'typ': 'telnet'})]
user_17405770

this would work:

my_list = [('admin', '', {'type': 'telnet'}), ('admin', '', {'type': 'telnet'})]

unique_list = []
for i in my_list:
    if i not in unique_list:
        unique_list.append(i)

print(unique_list)

result:

[('admin', '', {'type': 'telnet'})]
user_73181200

Any object being set as a dictionary key or a set item, needs to be hashable. hashing means that a python object, like a float number, string or tuple is converted to a unique number. This number allows easy lookup for the dictionary. However, it requires that the hashed element can't be changed, as this would also require the number to change.

Tuples themselves are hashable, as they can't be changed. However, they are only hashable if their content is also hashable. The strings are perfectly hashable, but dictionaries aren't. This means that you can't use dictionaries as dictionary keys.

Thus, the problem requires another solution. The simplest one would be to create a new list and then append every unique item in the list to it:

new_list = []

for x in my_list:
    if x not in new_list:
        new_list.append(x)

When you need the change to be in place, meaning that every variable assigned to that list also changes, you can add the following line:

my_list[:] = new_list
user_164906270

You cannot hash a dictionary, that is why you are getting an error. But you can convert it to a string, hash that string, use the fact that dictionary keys are unique, and recover their values (which have the original hashes).

lst = [('admin', '', {'type': 'telnet'}),
       ('admin', '', {'type': 'telnet'}),
       ('admin', "", {'type': 'telnet'}), ]

lst = list({str(x) : x for x in lst}.values())

print(lst)
# [('admin', '', {'type': 'telnet'})]
user_9676210
Related