Why can't dict be remove from a list of duplicate dicts

user_16252930 0
Viewed 74

I scraped data from a website but now the problem is that I can't remove duplicate dicts. The main problem is that regular ways are not working since some items contains dicts themselves.

[
{
    "content": {
        "link": "https://drive.google.com/open?id=1vVIhIn8Omz6fGmeriCDilOlKQhAOBItL",
        "title": "שיחו\"ק"
    },
    "id": "month_1_farbrengen_1_content",
    "event_title": "יום א' דר\"ה, בעת ההליכה לתשליך",
    "event_link": "https://www.mafteiach.app/all/5711-01-01"
},
{
    "content": {
        "link": "https://drive.google.com/file/d/1Q6o-wEkpAsvRQIaJqSqcCOkBZhQAAdYI/view?usp=drivesdk",
        "title": "תו\"מ התוועדויות"
    },
    "id": "month_1_farbrengen_1_content",
    "event_title": "יום א' דר\"ה, בעת ההליכה לתשליך",
    "event_link": "https://www.mafteiach.app/all/5711-01-01"
},
{
    "content": {
        "link": "https://drive.google.com/file/d/1V1sXWx_Jqbn2f_2P40v_DlqlHfXB1u-0/view?usp=sharing",
        "title": "Proceeding Together Vol. 3"
    },
    "id": "month_1_farbrengen_1_content",
    "event_title": "יום א' דר\"ה, בעת ההליכה לתשליך",
    "event_link": "https://www.mafteiach.app/all/5711-01-01"
},
{
    "content": {
        "link": "https://drive.google.com/file/d/1V1sXWx_Jqbn2f_2P40v_DlqlHfXB1u-0/view?usp=sharing",
        "title": "Proceeding Together Vol. 3"
    },
    "id": "month_1_farbrengen_1_content",
    "event_title": "יום א' דר\"ה, בעת ההליכה לתשליך",
    "event_link": "https://www.mafteiach.app/all/5711-01-01"
},
{
    "content": {
        "link": "https://drive.google.com/file/d/1V1sXWx_Jqbn2f_2P40v_DlqlHfXB1u-0/view?usp=sharing",
        "title": "Proceeding Together Vol. 3"
    },
    "id": "month_1_farbrengen_1_content",
    "event_title": "יום א' דר\"ה, בעת ההליכה לתשליך",
    "event_link": "https://www.mafteiach.app/all/5711-01-01"
},
{
    "content": {
        "link": "https://drive.google.com/file/d/1V1sXWx_Jqbn2f_2P40v_DlqlHfXB1u-0/view?usp=sharing",
        "title": "Proceeding Together Vol. 3"
    },
    "id": "month_1_farbrengen_1_content",
    "event_title": "יום א' דר\"ה, בעת ההליכה לתשליך",
    "event_link": "https://www.mafteiach.app/all/5711-01-01"
},
{
    "content": {
        "link": "https://drive.google.com/file/d/1V1sXWx_Jqbn2f_2P40v_DlqlHfXB1u-0/view?usp=sharing",
        "title": "Proceeding Together Vol. 3"
    },
    "id": "month_1_farbrengen_1_content",
    "event_title": "יום א' דר\"ה, בעת ההליכה לתשליך",
    "event_link": "https://www.mafteiach.app/all/5711-01-01"
},]

This is a samplde data. I have used all quite a few ways but all of them don't seem to work

beautifulsoup python
5 Answers

looking at the dicts they all have the same id but different links. so in this case it's better to remove the duplicates using link

my_dicts = [{'content': {'link': 'https://drive.google.com/open?id=1vVIhIn8Omz6fGmeriCDilOlKQhAOBItL', 'title': 'שיחו"ק'}, 'id': 'month_1_farbrengen_1_content', 'event_title': 'יום א\' דר"ה, בעת ההליכה לתשליך', 'event_link': 'https://www.mafteiach.app/all/5711-01-01'}, {'content': {'link': 'https://drive.google.com/file/d/1Q6o-wEkpAsvRQIaJqSqcCOkBZhQAAdYI/view?usp=drivesdk', 'title': 'תו"מ התוועדויות'}, 'id': 'month_1_farbrengen_1_content', 'event_title': 'יום א\' דר"ה, בעת ההליכה לתשליך', 'event_link': 'https://www.mafteiach.app/all/5711-01-01'}, {'content': {'link': 'https://drive.google.com/file/d/1V1sXWx_Jqbn2f_2P40v_DlqlHfXB1u-0/view?usp=sharing', 'title': 'Proceeding Together Vol. 3'}, 'id': 'month_1_farbrengen_1_content', 'event_title': 'יום א\' דר"ה, בעת ההליכה לתשליך', 'event_link': 'https://www.mafteiach.app/all/5711-01-01'}, {'content': {'link': 'https://drive.google.com/file/d/1V1sXWx_Jqbn2f_2P40v_DlqlHfXB1u-0/view?usp=sharing', 'title': 'Proceeding Together Vol. 3'}, 'id': 'month_1_farbrengen_1_content', 'event_title': 'יום א\' דר"ה, בעת ההליכה לתשליך', 'event_link': 'https://www.mafteiach.app/all/5711-01-01'}, {'content': {'link': 'https://drive.google.com/file/d/1V1sXWx_Jqbn2f_2P40v_DlqlHfXB1u-0/view?usp=sharing', 'title': 'Proceeding Together Vol. 3'}, 'id': 'month_1_farbrengen_1_content', 'event_title': 'יום א\' דר"ה, בעת ההליכה לתשליך', 'event_link': 'https://www.mafteiach.app/all/5711-01-01'}, {'content': {'link': 'https://drive.google.com/file/d/1V1sXWx_Jqbn2f_2P40v_DlqlHfXB1u-0/view?usp=sharing', 'title': 'Proceeding Together Vol. 3'}, 'id': 'month_1_farbrengen_1_content', 'event_title': 'יום א\' דר"ה, בעת ההליכה לתשליך', 'event_link': 'https://www.mafteiach.app/all/5711-01-01'}, {'content': {'link': 'https://drive.google.com/file/d/1V1sXWx_Jqbn2f_2P40v_DlqlHfXB1u-0/view?usp=sharing', 'title': 'Proceeding Together Vol. 3'}, 'id': 'month_1_farbrengen_1_content', 'event_title': 'יום א\' דר"ה, בעת ההליכה לתשליך', 'event_link': 'https://www.mafteiach.app/all/5711-01-01'}]
new_dict = []
n_urls = set()

for i in range(len(my_dicts)):
  if not my_dicts[i]['content']['link'] in n_urls:
    new_dict.append(my_dicts[i])
    n_urls.add(my_dicts[i]['content']['link'])
  
print(new_dict)

Output:

  [
   {'content': {'link': 'https://drive.google.com/open?id=1vVIhIn8Omz6fGmeriCDilOlKQhAOBItL',
   'title': 'שיחו"ק'},
  'event_link': 'https://www.mafteiach.app/all/5711-01-01',
  'event_title': 'יום א\' דר"ה, בעת ההליכה לתשליך',
  'id': 'month_1_farbrengen_1_content'},

   {'content': {'link': 'https://drive.google.com/file/d/1Q6o-wEkpAsvRQIaJqSqcCOkBZhQAAdYI/view?usp=drivesdk',
   'title': 'תו"מ התוועדויות'},
  'event_link': 'https://www.mafteiach.app/all/5711-01-01',
  'event_title': 'יום א\' דר"ה, בעת ההליכה לתשליך',
  'id': 'month_1_farbrengen_1_content'},

   {'content': {'link': 'https://drive.google.com/file/d/1V1sXWx_Jqbn2f_2P40v_DlqlHfXB1u-0/view?usp=sharing',
   'title': 'Proceeding Together Vol. 3'},
  'event_link': 'https://www.mafteiach.app/all/5711-01-01',
  'event_title': 'יום א\' דר"ה, בעת ההליכה לתשליך',
  'id': 'month_1_farbrengen_1_content'}]
user_110956440

If lst is your list from the question you can remove duplicates for example with:

out = []
while lst:
    item = lst.pop()
    if item not in out:
        out.append(item)

print(out)

Prints:

[
    {
        "content": {
            "link": "https://drive.google.com/file/d/1V1sXWx_Jqbn2f_2P40v_DlqlHfXB1u-0/view?usp=sharing",
            "title": "Proceeding Together Vol. 3",
        },
        "id": "month_1_farbrengen_1_content",
        "event_title": "יום א' דר\"ה, בעת ההליכה לתשליך",
        "event_link": "https://www.mafteiach.app/all/5711-01-01",
    },
    {
        "content": {
            "link": "https://drive.google.com/file/d/1Q6o-wEkpAsvRQIaJqSqcCOkBZhQAAdYI/view?usp=drivesdk",
            "title": 'תו"מ התוועדויות',
        },
        "id": "month_1_farbrengen_1_content",
        "event_title": "יום א' דר\"ה, בעת ההליכה לתשליך",
        "event_link": "https://www.mafteiach.app/all/5711-01-01",
    },
    {
        "content": {
            "link": "https://drive.google.com/open?id=1vVIhIn8Omz6fGmeriCDilOlKQhAOBItL",
            "title": 'שיחו"ק',
        },
        "id": "month_1_farbrengen_1_content",
        "event_title": "יום א' דר\"ה, בעת ההליכה לתשליך",
        "event_link": "https://www.mafteiach.app/all/5711-01-01",
    },
]
user_100359850

first we create a newItems array. then we loop through our items and everytime check if current item is in the newItems array or not. if its not there we simple add it to newItems.

newItems = []

for item in items:
    if item not in newItems:
        newItems.append(item)
user_117845500

Since the format of your dictionary looks fixed, you can wrap it a little:

>>> class Foo:
...     def __init__(self, dct):
...         self.dct = dct
...         self.link = dct['content']['link']
...     def __hash__(self):
...         return hash(self.link)
...     def __eq__(self, other):
...         return self.link == other.link if isinstance(other, Foo) else False
...
>>> [f.dct for f in {Foo(dct) for dct in dict_list}]
[{'content': {'link': 'https://drive.google.com/open?id=1vVIhIn8Omz6fGmeriCDilOlKQhAOBItL',
              'title': 'שיחו"ק'},
  'id': 'month_1_farbrengen_1_content',
  'event_title': 'יום א\' דר"ה, בעת ההליכה לתשליך',
  'event_link': 'https://www.mafteiach.app/all/5711-01-01'},
 {'content': {'link': 'https://drive.google.com/file/d/1Q6o-wEkpAsvRQIaJqSqcCOkBZhQAAdYI/view?usp=drivesdk',
              'title': 'תו"מ התוועדויות'},
  'id': 'month_1_farbrengen_1_content',
  'event_title': 'יום א\' דר"ה, בעת ההליכה לתשליך',
  'event_link': 'https://www.mafteiach.app/all/5711-01-01'},
 {'content': {'link': 'https://drive.google.com/file/d/1V1sXWx_Jqbn2f_2P40v_DlqlHfXB1u-0/view?usp=sharing',
              'title': 'Proceeding Together Vol. 3'},
  'id': 'month_1_farbrengen_1_content',
  'event_title': 'יום א\' דר"ה, בעת ההליכה לתשליך',
  'event_link': 'https://www.mafteiach.app/all/5711-01-01'}]
user_179809310 
[i for i in l if l.count(i)==1 ]
user_52264700
Related