I have a list "records" like this
data = [
{'id':1, 'name': 'A', 'price': 10, 'url': 'foo'},
{'id':2, 'name': 'A', 'price': 20, 'url': 'bar'},
{'id':3, 'name': 'A', 'price': 30, 'url': 'baz'},
{'id':4, 'name': 'A', 'price': 10, 'url': 'baz'},
{'id':5, 'name': 'A', 'price': 20, 'url': 'bar'},
{'id':6, 'name': 'A', 'price': 30, 'url': 'foo'},
{'id':7, 'name': 'A', 'price': 99, 'url': 'quu'},
{'id':8, 'name': 'B', 'price': 10, 'url': 'foo'},
]
I want to remove records that are "duplicates", where equality is defined by a list of logical conditions. Each element in the list is an OR condition, and all elements are ANDed together. For example:
filters = [ ['name'], ['price', 'url'] ]
means that two records are considered equal if their name AND (their price OR url) are equal. For the above example:
For item 1 the duplicates are 4 (by name and price) and 6 (name+url)
For item 2 - 5 (name+price, name+url)
For item 3 - 4 (name+url) and 6 (name+price)
For item 7 there are no duplicates (neither price nor url match)
For item 8 there are no duplicates (name doesn't match)
So the resulting list must contain items 1, 2, 3, 7 and 8.
Please take into account that
- there might be more AND conditions:
['name'], ['price', 'url'], ['weight'], ['size'], ... - the OR groups in the conditions list can be longer than 2 items, e.g.
['name'], ['price', 'url', 'weight']... - the source list is very long, an
O(n^2)alogirthm is out of the question