What's the pythonic way to remove element from a list of dict if ONE of the values is not unique?

Here is an approach using any() and list-comprehension:

Code:

l=[
    {'a': 1, 'b': 'j'},
    {'a': 2, 'b': 'j'},
    {'a': 3, 'b': 'i'}
]

new_l = []

for d in l:
    if any([d['b'] == x['b'] for x in new_l]):
        continue
    new_l.append(d)

print(new_l)

Output:

[{'a': 1, 'b': 'j'}, {'a': 3, 'b': 'i'}]

def drop_dup_key(src, key):
    ''' src is the source list, and key is a function to obtain the key'''
    keyset, result = set(), []
    for elem in src:
        keyval = key(elem)
        if keyval not in keyset:
             result.append(elem)
             keyset.add(keyval)
    return result

Use it like this:

drop_dup_key(in_list, lambda d: return d.get('b'))

You could define a custom container class which implements the __eq__ and __hash__ magic methods. That way, you can use a set to remove "duplicates" (according to your criteria). This doesn't necessarily preserve order.

from itertools import starmap
from typing import NamedTuple

class MyTuple(NamedTuple):
    a: int
    b: str

    def __eq__(self, other):
        return self.b == other.b

    def __hash__(self):
        return ord(self.b)


print(set(starmap(MyTuple, [(1, 'j'), (2, 'j'), (3, 'i')])))

Output:

{MyTuple(a=3, b='i'), MyTuple(a=1, b='j')}
>>> 

I suggest this implementation:

_missing = object()
def dedupe(iterable, selector=_missing):
    "De-duplicate a sequence based on a selector"
    keys = set()
    if selector is _missing: selector = lambda e: e
    for e in iterable:
        if selector(e) in keys: continue
        keys.add(selector(e))
        yield e

Advantages:

• Returns a generator:
  It iterates the original collection just once, lazily. That could be useful and/or performatic in some scenarios, specially if you will chain additional query operations.

  input = [{'a': 1, 'b': 'j'}, {'a': 2, 'b': 'j'}, {'a': 3, 'b': 'i'}]
  s = dedupe(input, lambda x: x['b'])
  s = map(lambda e: e['a'], s)
  sum(s) # Only now the list is iterated. Result: 4
  

• Accepts any kind of iterable:
  Be it a list, set, dictionary or a custom iterable class. You can construct whatever collection type out of it, without iterating multiple times.

  d = {'a': 1, 'b': 1, 'c': 2}
  
  {k: v for k, v in dedupe(d.items(), lambda e: e[1])}
  # Result (dict): {'a': 1, 'c': 2}
  
  {*dedupe(d.items(), lambda e: e[1])}
  # Result (set of tuples): {('a', 1), ('c', 2)}
  

• Takes an optional selector function (or any callable):
  This gives you flexibility to re-use this function in many different contexts, with any custom logic or types. If the selector is absent, it compares the whole elements.

  # de-duping based on absolute value:
  (*dedupe([-3, -2, -2, -1, 0, 1, 1, 2, 3, 3], abs),)
  # Result: (-3, -2, -1, 0)
  
  # de-duping without selector:
  (*dedupe([-3, -2, -2, -1, 0, 1, 1, 2, 3, 3]),)
  # Result: (-3, -2, -1, 0, 1, 2, 3)
  

The comparison of tuples to dictionaries isn't quite accurate since the tuples only contain the dictionary values, not the keys, and I believe you are asking about duplicate key:value pairs.

Here is a solution which I believe solves your problem, but might not be as pythonic as possible.

seen = set()
kept = []

for d in x:
     keep = True
     for k, v in d.items():
         if (k, v) in seen:
             keep = False
             break
         seen.add((k, v))
     if keep:
         kept.append(d)

print(kept)

Output:

[{'a': 1, 'b': 'j'}, {'a': 3, 'b': 'i'}]