How to remove items from a list and their associated ones in other lists

Viewed 74

I'm working on a project, and I got to the point where I need to remove any duplicates from the main list. I've got three lists here, and I'm trying to eliminate duplicates in the flight_ID list. I managed to do it, but unfortunately, I couldn't remove other elements that are associated with the removed ones in the flight_ID list.

# All lists have a length of 20

flight_ID = ['1064662221', '1064617390', '1064614152', '1064614152', \
 '1064775880', '1064645826', '1064645826', '1064664535', '1064659772', \
 '1064659772', '1064614050', '1064614050', '1064614286', '1064614286', \
'1064614286', '1064614286', '1064614286', '1064614286', '1064614286', '1064646536']

flight_number = ['1827', '1585', '8409', '1465', '30', '9188', '2232', '3760', '579', '3309', '1259', '2193', '6566', '2231', '5214', '8601', '3169', '1601', '7832', '335']

airline_Code = ['TK', 'AY', 'DL', 'AF', 'FX', 'UA', 'LH', 'U2', 'SK', 'A3', 'AF', 'KL', 'VS', 'UX', 'G3', 'UU', 'KQ', 'AF', 'AR', 'LO']

I used the following function to remove duplicates from the main list:

def remove_dup(a):
   i = 0
   while i < len(a):
      j = i + 1
      while j < len(a):
         if a[i] == a[j]:
            del a[j]
         else:
            j += 1
      i += 1

remove_dup(flight_ID)

# OUTPUT
['1064662221', '1064617390', '1064614152', '1064775880', '1064645826', '1064664535', '1064659772', '1064614050', '1064614286', '1064646536']

# 10 elements have been removed.

Now, as I described above, I need to do the same thing with the other lists, so Items matching items in the main list (flight_ID) are also removed.

NOTE: Although the main list shows duplicate items, other lists' items DON'T

5 Answers

I'd suggest using Pandas if you're going to do more with data formatted in the way you described as it makes operations like removing duplicates possible in a pain-free manner:

import pandas as pd

# Make a DataFrame
flight_ID = ['1064662221', '1064617390', ...]
flight_number = ['1827', '1585', '8409', ...]
airline_Code = ['TK', 'AY', 'DL', ...]

df = pd.DataFrame({'flight_ID': flight_ID,
                   'flight_number': flight_number,
                   'airline_Code': airline_Code})

# Remove duplicates - just one line!
df.drop_duplicates('flight_ID', inplace=True)

You get a DataFrame that looks like this:

     flight_ID flight_number airline_Code
0   1064662221          1827           TK
1   1064617390          1585           AY
2   1064614152          8409           DL
4   1064775880            30           FX
5   1064645826          9188           UA
7   1064664535          3760           U2
8   1064659772           579           SK
10  1064614050          1259           AF
12  1064614286          6566           VS
19  1064646536           335           LO

First, alter your representation to link the items as needed, rather than using parallel lists.

flight_list = zip(flight_ID, flight_number, airline_Code)

This makes it much easier to drop the three related items.

Now, use any of the standard ways of dropping duplicates. In each one build a new list: altering a your iteration target is a bad idea, as documented in many posts on this site. Keeping this at your demonstrated programming level:

unique_flight = []
found_ID = set()
for flight in flight_list:
    if flight[0] not in found_ID:
        found_ID.add(flight[0])
        unique_flight.append(flight)

for flight in unique_flight:
    print(flight)

Output:

('1064662221', '1827', 'TK')
('1064617390', '1585', 'AY')
('1064614152', '8409', 'DL')
('1064775880', '30', 'FX')
('1064645826', '9188', 'UA')
('1064664535', '3760', 'U2')
('1064659772', '579', 'SK')
('1064614050', '1259', 'AF')
('1064614286', '6566', 'VS')
('1064646536', '335', 'LO')

Here are a few ways, but I would think about using a class to represent this kind of data (similar to how the namedtuple example works)

Adding the flight_IDs to a dictionary as the key makes them unique, with values as the indexes:

flight_ID_inds = {f: i for i, f in enumerate(flight_ID)}
flight_ID = list(flight_ID_inds.keys())
flight_number = [flight_number[i] for i in flight_ID_inds.values()]
airline_Code = [airline_Code[i] for i in flight_ID_inds.values()]

Again, but with values as a tuple for the other list datas instead of the index:

dic = {fid: (fn, ac) for fid, fn, ac in zip(flight_ID, flight_number, airline_Code)}
flight_ID = list(dic.keys())
flight_number = [x[0] for x in dic.values()]
airline_Code = [x[1] for x in dic.values()]

Using a named tuple (using a list-of-dicts representation would also work):

from collections import namedtuple

flight_nt = namedtuple("Flight", "flight_ID, flight_number, airline_Code")

flights = [flight_nt(fid, fn, ac) for fid, fn, ac in zip(flight_ID, flight_number, airline_Code)]
uniq_ids = set()
uniq_flights = []
for f in flights:
    if f.flight_ID not in uniq_ids:
        uniq_ids.add(f.flight_ID)
        uniq_flights.append(f)
flight_ID = [x.flight_ID for x in uniq_flights]
flight_number = [x.flight_number for x in uniq_flights]
airline_Code = [x.airline_Code for x in uniq_flights]

I would recommend a object oriented (class or dataclass) for this kind problem:

class Flight:
    def __init__(self, flight_id, flight_number, airline_code):
        self.flight_id = flight_id
        self.flight_number = flight_number
        self.airline_code = airline_code

    def __hash__(self):
        return hash(self.flight_id)

    def __eq__(self, other):
        return other.flight_id == self.flight_id

flights = [Flight(fid, fn, ac) for fid, fn, ac in zip(flight_ID, flight_number, airline_Code)]
uniq_flights = set(flights)

@Prune has a better solution, but you could always use enumerate()

for index, id in enumerate(flight_ID):
    if id in flight_ID[index:]:
        del flight_ID[index]
        del flight_number[index]
        del airline_Code[index]

Note this doesn't preserve order, if you wanted to do that you'll have to find the index of the value in the slice.

You can first determine the ones to keep/remove and then use itertools.compress to remove the elements:

import itertools as it

keep = []
seen = set()
for x in flight_ID:
    keep.append(x not in seen)
    seen.add(x)

flight_ID = list(it.compress(flight_ID, keep))
flight_number = list(it.compress(flight_number, keep))
airline_Code = list(it.compress(airline_Code, keep))

However since this data seems to logically belong together, perhaps it would be a good idea to create a dedicated container class for it, e.g. via namedtuple:

from collections import namedtuple

FlighData = namedtuple('id number code')

data = [FlightData(*x) for x in zip(flight_ID, flight_number, airline_Code)]

Then another way would be to use itertools.groupby:

unique_data = list(next(g) for k, g in it.groupby(sorted(data), key=op.itemgetter(0)))
Related