Creating a new List out of a List but without dublicate items?

Question

i have a List

carner_list = ['Damselfly',
 'Damselfly',
 'Damselfly',
 'The Isle of Arran',
 'Mean It in the Morning']

Now i want to get rid of the dublicated items. The Problem is, the items that are double are only different from each other at a specific point in the string, i[38:].

My idea was to create a for loop:

new_list = []
for i in carner_list:
       if i[38:] in new_list:
           print("found")
       else:
           new_list = new_list + [i]
           print("not")

But this is not working.

Is something in the syntax wrong or am i completely on the wrong track?

Best Russell

user_14190365 · Answer

I typed a small function called listContains that I think can solve your problem. Your code is not working because you search for the value i[38:] in your new_list, when in new_list you append the entire value of i.
So you should also applied the rule of [38:] on each value of the list.
I think the code below can explain better what I'm saying:

carner_list = ['Damselfly',
 'Damselfly',
 'Damselfly',
 'The Isle of Arran',
 'Mean It in the Morning']
new_list = []

def listContains(myList, toSearch):
  for val in myList:
    if val[38:] == toSearch:
      return True
  return False

for i in carner_list:
  if listContains(new_list, i[38:]):
    print("found")
  else:
    new_list.append(i)
    print("not")
print(new_list)

If you want to test it you can do it from here

user_6890912 · Answer

The portion of the strings (from index 38 to the end) with which you use to determine duplication is not what you actually store in the list, so the in operator won't work.

You can instead use a dict to store the de-duplicated strings, with the portion of the strings you care about as the key, so that the in operator can work:

new = {}
for i in carner_list:
    key = i[38:]
    if key not in new:
        new[key] = i
print(list(new.values()))

This outputs:

['Damselfly', 'The Isle of Arran', 'Mean It in the Morning']

user_5003873 · Answer

So the way you are currently searching, you are looking if the sub string is equal to anything in the new_list. This will never be true since it's a sub string.

You can use a lambda and then filter it for true results to see if the item is in the new list. Then cast it to a list and check if that list has a length not equal to 0.

len(list(filter(lambda x: i[38:] in x, new_list))) != 0

Final Code

carner_list = ['Damselfly',
 'Damselfly',
 'Damselfly',
 'The Isle of Arran',
 'Mean It in the Morning']


new_list = []

for i in carner_list:
    if len(list(filter(lambda x: i[38:] in x, new_list))) != 0:
        print("found")
    else:
        new_list.append(i)
        print("not")

user_532312 · Answer

Use BeautifulSoup to parse the html and then check

Ex:

from bs4 import BeautifulSoup

carner_list = ['Damselfly',
 'Damselfly',
 'Damselfly',
 'The Isle of Arran',
 'Mean It in the Morning']

new_list = []
check_val = set()
for i in carner_list:
    s = BeautifulSoup(i, "html.parser")
    if s.text not in check_val:    #check for text
        new_list.append(i)
        check_val.add(s.text)
print(new_list)

Output:

['Damselfly',
 'The Isle of '
 'Arran',
 'Mean It in the '
 'Morning']

user_11622508 · Answer

Why not use regular expressions

import re
carner_list = ['Damselfly',
 'Damselfly',
 'Damselfly',
 'The Isle of Arran',
 'Mean It in the Morning']

print({re.findall(r'"([^"]*)"', x)[0].split("/")[4]: x for x in carner_list })

#Below is the output generated 
'''
{'Damselfly': 'Damselfly', 'The+Isle+of+Arran': 'The Isle of Arran', 'Mean+It+in+the+Morning': 'Mean It in the Morning'}
'''

Creating a new List out of a List but without dublicate items?

5 Answers