How to remove items from a list while iterating?

Viewed 786279

I'm iterating over a list of tuples in Python, and am attempting to remove them if they meet certain criteria.

for tup in somelist:
    if determine(tup):
         code_to_remove_tup

What should I use in place of code_to_remove_tup? I can't figure out how to remove the item in this fashion.

25 Answers

You can use a list comprehension to create a new list containing only the elements you don't want to remove:

somelist = [x for x in somelist if not determine(x)]

Or, by assigning to the slice somelist[:], you can mutate the existing list to contain only the items you want:

somelist[:] = [x for x in somelist if not determine(x)]

This approach could be useful if there are other references to somelist that need to reflect the changes.

Instead of a comprehension, you could also use itertools. In Python 2:

from itertools import ifilterfalse
somelist[:] = ifilterfalse(determine, somelist)

Or in Python 3:

from itertools import filterfalse
somelist[:] = filterfalse(determine, somelist)

The answers suggesting list comprehensions are ALMOST correct -- except that they build a completely new list and then give it the same name the old list as, they do NOT modify the old list in place. That's different from what you'd be doing by selective removal, as in @Lennart's suggestion -- it's faster, but if your list is accessed via multiple references the fact that you're just reseating one of the references and NOT altering the list object itself can lead to subtle, disastrous bugs.

Fortunately, it's extremely easy to get both the speed of list comprehensions AND the required semantics of in-place alteration -- just code:

somelist[:] = [tup for tup in somelist if determine(tup)]

Note the subtle difference with other answers: this one is NOT assigning to a barename - it's assigning to a list slice that just happens to be the entire list, thereby replacing the list contents within the same Python list object, rather than just reseating one reference (from previous list object to new list object) like the other answers.

You need to take a copy of the list and iterate over it first, or the iteration will fail with what may be unexpected results.

For example (depends on what type of list):

for tup in somelist[:]:
    etc....

An example:

>>> somelist = range(10)
>>> for x in somelist:
...     somelist.remove(x)
>>> somelist
[1, 3, 5, 7, 9]

>>> somelist = range(10)
>>> for x in somelist[:]:
...     somelist.remove(x)
>>> somelist
[]
for i in range(len(somelist) - 1, -1, -1):
    if some_condition(somelist, i):
        del somelist[i]

You need to go backwards otherwise it's a bit like sawing off the tree-branch that you are sitting on :-)

Python 2 users: replace range by xrange to avoid creating a hardcoded list

Your best approach for such an example would be a list comprehension

somelist = [tup for tup in somelist if determine(tup)]

In cases where you're doing something more complex than calling a determine function, I prefer constructing a new list and simply appending to it as I go. For example

newlist = []
for tup in somelist:
    # lots of code here, possibly setting things up for calling determine
    if determine(tup):
        newlist.append(tup)
somelist = newlist

Copying the list using remove might make your code look a little cleaner, as described in one of the answers below. You should definitely not do this for extremely large lists, since this involves first copying the entire list, and also performing an O(n) remove operation for each element being removed, making this an O(n^2) algorithm.

for tup in somelist[:]:
    # lots of code here, possibly setting things up for calling determine
    if determine(tup):
        newlist.append(tup)

For those that like functional programming:

somelist[:] = filter(lambda tup: not determine(tup), somelist)

or

from itertools import ifilterfalse
somelist[:] = list(ifilterfalse(determine, somelist))

Most of the answers here want you to create a copy of the list. I had a use case where the list was quite long (110K items) and it was smarter to keep reducing the list instead.

First of all you'll need to replace foreach loop with while loop,

i = 0
while i < len(somelist):
    if determine(somelist[i]):
         del somelist[i]
    else:
        i += 1

The value of i is not changed in the if block because you'll want to get value of the new item FROM THE SAME INDEX, once the old item is deleted.

This answer was originally written in response to a question which has since been marked as duplicate: Removing coordinates from list on python

There are two problems in your code:

1) When using remove(), you attempt to remove integers whereas you need to remove a tuple.

2) The for loop will skip items in your list.

Let's run through what happens when we execute your code:

>>> L1 = [(1,2), (5,6), (-1,-2), (1,-2)]
>>> for (a,b) in L1:
...   if a < 0 or b < 0:
...     L1.remove(a,b)
... 
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
TypeError: remove() takes exactly one argument (2 given)

The first problem is that you are passing both 'a' and 'b' to remove(), but remove() only accepts a single argument. So how can we get remove() to work properly with your list? We need to figure out what each element of your list is. In this case, each one is a tuple. To see this, let's access one element of the list (indexing starts at 0):

>>> L1[1]
(5, 6)
>>> type(L1[1])
<type 'tuple'>

Aha! Each element of L1 is actually a tuple. So that's what we need to be passing to remove(). Tuples in python are very easy, they're simply made by enclosing values in parentheses. "a, b" is not a tuple, but "(a, b)" is a tuple. So we modify your code and run it again:

# The remove line now includes an extra "()" to make a tuple out of "a,b"
L1.remove((a,b))

This code runs without any error, but let's look at the list it outputs:

L1 is now: [(1, 2), (5, 6), (1, -2)]

Why is (1,-2) still in your list? It turns out modifying the list while using a loop to iterate over it is a very bad idea without special care. The reason that (1, -2) remains in the list is that the locations of each item within the list changed between iterations of the for loop. Let's look at what happens if we feed the above code a longer list:

L1 = [(1,2),(5,6),(-1,-2),(1,-2),(3,4),(5,7),(-4,4),(2,1),(-3,-3),(5,-1),(0,6)]
### Outputs:
L1 is now: [(1, 2), (5, 6), (1, -2), (3, 4), (5, 7), (2, 1), (5, -1), (0, 6)]

As you can infer from that result, every time that the conditional statement evaluates to true and a list item is removed, the next iteration of the loop will skip evaluation of the next item in the list because its values are now located at different indices.

The most intuitive solution is to copy the list, then iterate over the original list and only modify the copy. You can try doing so like this:

L2 = L1
for (a,b) in L1:
    if a < 0 or b < 0 :
        L2.remove((a,b))
# Now, remove the original copy of L1 and replace with L2
print L2 is L1
del L1
L1 = L2; del L2
print ("L1 is now: ", L1)

However, the output will be identical to before:

'L1 is now: ', [(1, 2), (5, 6), (1, -2), (3, 4), (5, 7), (2, 1), (5, -1), (0, 6)]

This is because when we created L2, python did not actually create a new object. Instead, it merely referenced L2 to the same object as L1. We can verify this with 'is' which is different from merely "equals" (==).

>>> L2=L1
>>> L1 is L2
True

We can make a true copy using copy.copy(). Then everything works as expected:

import copy
L1 = [(1,2), (5,6),(-1,-2), (1,-2),(3,4),(5,7),(-4,4),(2,1),(-3,-3),(5,-1),(0,6)]
L2 = copy.copy(L1)
for (a,b) in L1:
    if a < 0 or b < 0 :
        L2.remove((a,b))
# Now, remove the original copy of L1 and replace with L2
del L1
L1 = L2; del L2
>>> L1 is now: [(1, 2), (5, 6), (3, 4), (5, 7), (2, 1), (0, 6)]

Finally, there is one cleaner solution than having to make an entirely new copy of L1. The reversed() function:

L1 = [(1,2), (5,6),(-1,-2), (1,-2),(3,4),(5,7),(-4,4),(2,1),(-3,-3),(5,-1),(0,6)]
for (a,b) in reversed(L1):
    if a < 0 or b < 0 :
        L1.remove((a,b))
print ("L1 is now: ", L1)
>>> L1 is now: [(1, 2), (5, 6), (3, 4), (5, 7), (2, 1), (0, 6)]

Unfortunately, I cannot adequately describe how reversed() works. It returns a 'listreverseiterator' object when a list is passed to it. For practical purposes, you can think of it as creating a reversed copy of its argument. This is the solution I recommend.

If you want to delete elements from a list while iterating, use a while-loop so you can alter the current index and end index after each deletion.

Example:

i = 0
length = len(list1)

while i < length:
    if condition:
        list1.remove(list1[i])
        i -= 1
        length -= 1

    i += 1

for loop will be iterate through index..

consider you have a list,

[5, 7, 13, 29, 65, 91]

you have using list variable called lis. and you using same to remove..

your variable

lis = [5, 7, 13, 29, 35, 65, 91]
       0  1   2   3   4   5   6

during 5th iteration,

your number 35 was not a prime so you removed it from a list.

lis.remove(y)

and then next value (65) move on to previous index.

lis = [5, 7, 13, 29, 65, 91]
       0  1   2   3   4   5

so 4th iteration done pointer moved onto 5th..

thats why your loop doesnt cover 65 since its moved into previous index.

so you shouldn't reference list into another variable which still reference original instead of copy.

ite = lis #dont do it will reference instead copy

so do copy of list using list[::]

now you it will give,

[5, 7, 13, 29]

Problem is you removed a value from a list during iteration then your list index will collapse.

so you can try comprehension instead.

which supports all the iterable like, list, tuple, dict, string etc

You might want to use filter() available as the built-in.

For more details check here

The most effective method is list comprehension, many people show their case, of course, it is also a good way to get an iterator through filter.

Filter receives a function and a sequence. Filter applies the passed function to each element in turn, and then decides whether to retain or discard the element depending on whether the function return value is True or False.

There is an example (get the odds in the tuple):

list(filter(lambda x:x%2==1, (1, 2, 4, 5, 6, 9, 10, 15)))  
# result: [1, 5, 9, 15]

Caution: You can also not handle iterators. Iterators are sometimes better than sequences.

In some situations, where you're doing more than simply filtering a list one item at time, you want your iteration to change while iterating.

Here is an example where copying the list beforehand is incorrect, reverse iteration is impossible and a list comprehension is also not an option.

""" Sieve of Eratosthenes """

def generate_primes(n):
    """ Generates all primes less than n. """
    primes = list(range(2,n))
    idx = 0
    while idx < len(primes):
        p = primes[idx]
        for multiple in range(p+p, n, p):
            try:
                primes.remove(multiple)
            except ValueError:
                pass #EAFP
        idx += 1
        yield p

I can think of three approaches to solve your problem. As an example, I will create a random list of tuples somelist = [(1,2,3), (4,5,6), (3,6,6), (7,8,9), (15,0,0), (10,11,12)]. The condition that I choose is sum of elements of a tuple = 15. In the final list we will only have those tuples whose sum is not equal to 15.

What I have chosen is a randomly chosen example. Feel free to change the list of tuples and the condition that I have chosen.

Method 1.> Use the framework that you had suggested (where one fills in a code inside a for loop). I use a small code with del to delete a tuple that meets the said condition. However, this method will miss a tuple (which satisfies the said condition) if two consecutively placed tuples meet the given condition.

for tup in somelist:
    if ( sum(tup)==15 ): 
        del somelist[somelist.index(tup)]

print somelist
>>> [(1, 2, 3), (3, 6, 6), (7, 8, 9), (10, 11, 12)]

Method 2.> Construct a new list which contains elements (tuples) where the given condition is not met (this is the same thing as removing elements of list where the given condition is met). Following is the code for that:

newlist1 = [somelist[tup] for tup in range(len(somelist)) if(sum(somelist[tup])!=15)]

print newlist1
>>>[(1, 2, 3), (7, 8, 9), (10, 11, 12)]

Method 3.> Find indices where the given condition is met, and then use remove elements (tuples) corresponding to those indices. Following is the code for that.

indices = [i for i in range(len(somelist)) if(sum(somelist[i])==15)]
newlist2 = [tup for j, tup in enumerate(somelist) if j not in indices]

print newlist2
>>>[(1, 2, 3), (7, 8, 9), (10, 11, 12)]

Method 1 and method 2 are faster than method 3. Method2 and method3 are more efficient than method1. I prefer method2. For the aforementioned example, time(method1) : time(method2) : time(method3) = 1 : 1 : 1.7

If you will use the new list later, you can simply set the elem to None, and then judge it in the later loop, like this

for i in li:
    i = None

for elem in li:
    if elem is None:
        continue

In this way, you dont't need copy the list and it's easier to understand.

For anything that has the potential to be really big, I use the following.

import numpy as np

orig_list = np.array([1, 2, 3, 4, 5, 100, 8, 13])

remove_me = [100, 1]

cleaned = np.delete(orig_list, remove_me)
print(cleaned)

That should be significantly faster than anything else.

Related