Cleaning list to remove semi-duplicate values

user_10893983 0
Viewed 106

I have a list of of video links. Some of these links are almost duplicates, meaning they contain almost the same link except that it has x_480.mp4 instead of x.mp4. Not all links have those "_480" at the end.

How can I clean the list to get only the ones that end with _480.mp4, removing their alternate versions, and keep the ones without a _480.mp4 version?

Example:

videos=["VfeHB0sga.mp4","G9uKZiNm.mp4","VfeHB0sga_480.mp4","kvlX4Fa4.mp4"]

Expected result:

["G9uKZiNm.mp4","VfeHB0sga_480.mp4","kvlX4Fa4.mp4"]`

Note: all links ends with .mp4. Also, there are no _480.mp4 without original one.

By the way len(videos) is 243.

python-3.x list python
4 Answers

You can do it in two lines of code:

to_remove = {fn[:-8] + '.mp4' for fn in videos if fn.endswith('_480.mp4')}
cleaned = [fn for fn in videos if fn not in to_remove]

The first line uses a set comprehension to extract all of the _480.mp4 filenames, converting them to their unwanted short versions. They are stored in a set for quick searching.

The second line uses a list comprehension to filter out the unwanted filenames.

user_6755680

I'd probably go the dict route to not have to check for existence of items in a list (would become a (performance) problem for large lists). For instance:

list({v[:-8] if v.endswith("_480.mp4") else v[:-4]: v
      for v in sorted(videos)}.values())

That is compact way to say.

Construct me a dictionary whose key is incoming v without last 8 characters for values ending with "_480.mp4" or otherwise just stripped of last four character and being assigned value of the full incoming string.

Give me just values of that dictionary and since input was a list, I've passed it to a list constructor to get the same type as output.

Or broken down for easier reading, it could look something like this:

videos=["x.mp4","y.mp4","z.mp4","x_480.mp4"]

video_d = {}

for video_name in sorted(videos):
    if video_name.endswith("_480.mp4"):
        video_d[video_name[:-8]] = video_name
    else:
        video_d[video_name[:-4]] = video_name

new_videos = list(video_d.values())

It uses a virtual base name (stripping _480.mp4 or .mp4) as dictionary key. Since you do not care about resulting list order, we've made sure _480 suffixed entries are sorted after the "plain" entries. That way if they appear, they overwrite keys created for values without _480 suffix.

user_92882000

This should work. It loops through the videos until it finds one which ends with "_480.mp4". It then splits the title and get the starting bit and add ".mp4" to to create the video title which you want to remove. It then loops through the videos again and removes the video with that title.

videos=["x.mp4","y.mp4","z.mp4","x_480.mp4"]

#Loops through all the videos
for video in videos:
    if "_480.mp4" in video:
        #Removes the "_480" part of the video title
        start = video.replace("_480", "")
        for video2 in videos:
            if video2 == start:
                videos.remove(start)

print(videos)
user_53287330

You can even do this with one liner list comprehension.

[x for x in videos if x.split('.')[0] + '_480.mp4' not in videos]
user_65982750
Related