How to make use of an Iterator multiple times in python

user_4932296 0
Viewed 852

I am working on a task that requires that I use an iterator multiple times. For example

   #data
   fruit= ("grape", "banana", "apple")
   #iterator
   myit = iter(fruit)

   #the function I have
   def printIter(its):
     for x in its:
        print(x)

   def printIter2(its):
     for x in its:
        print(x)

I have to call printIter on the iterator twice but it is to perform completely different functions. But an iterator can only be consumed once. I don't have control over the data source fruit and iterator myit. I only have control over the functions printIter().

How best can I achieve my aim using less memory.

What i currently have:

   it1, it2 = itertools.tee(its)
   printIter(it1)
   printIter(it2)
   del it1, it2

Is this a good practice, any other way?

iterable iterator python-3.x python loops
2 Answers

If all you have is an iterator and you need to do two kinds of processing on it without consuming too much memory, your best bet is to design the processing you're doing to work in parallel. That is, you need to be able to do both parts of your processing on one item at a time. In your example, both of your iterator-consuming functions were just printing it out, which doesn't lend it self well to parallelization (you'd get the printout in a different order, e.g. 1, 1, 2, 2, 3, 3, ...). But for other kinds of problems, it's easy to do part of the work and then wait for more data.

Here's an example where I use two generator functions to consume a tee'd iterator in parallel (using the builtin zip). One adds up the values it gets and prints only the final sum, and the other prints them individually.

def consume1(it):
    total = 0
    for value in it:
        total += value
        yield
    print(total)

def consume2(it):
    for value in it:
        print(value)
        yield

opaque_iterator = iter((1, 2, 3, 4))
it1, it2 = itertools.tee(opaque_iterator)

for _ in zip(consume1(it1), consume2(it2)):
    pass

Output:

1
2
3
4
10

There are a bunch of subtleties to this kind of code, so don't be surprised if you don't get it working in your first attempt. My code above is pretty fragile, as zip isn't really designed for managing separate generators like this.

user_14050650

Since iterators are stateful and its resources consumed I am not sure what the goal of using the same iterator twice would be.

However, if you dont want to have the memory overhead of two copies of the iterator at the same time as is happening with tee(), you can just redeclare the iterator after the first is consumed and deleted

import itertools

#the function I have
def printIter(its):
  for x in its:
    print(x)

def printIter2(its):
  for x in its:
    print(x)


#data
fruit= ("grape", "banana", "apple")

#iterator
myit = iter(fruit)

#it1, it2 = itertools.tee(myit)
printIter(myit)
del myit
myit = iter(fruit)
printIter2(myit)
del myit

Since you indicated that you have no access to the original data, tee() is probably the best you can do with iterators. However, you can consider converting the single iterator to a list, then do the repetitive operations on that. 

import itertools
#data
fruit= ("grape", "banana", "apple")
#iterator
myit = iter(fruit)

def printIter(its):
  for x in its:
    print(x)

mylist = list(myit)
del myit
printIter(mylist)
printIter(mylist)
user_54784620
Related