Is there a Python standard library function to create a generator from repeatedly calling a functional?

user_4412981 0
Viewed 68

I have a method want to call repeatedly to iterate over, which will raise a StopIteration when it's done (in this case an instance of pyarrow.csv.CSVStreamingReader looping over a large file). I can use it in a for loop like this:

def batch_generator():
    while True:
        try:
            yield reader.read_next_batch()
        except StopIteration:
            return

for batch in batch_generator():
    writer.write_table(batch)

It can be done in a generic way with a user-defined function:

def make_generator(f):
    def gen():
        while True:
            try:
                yield f()
            except StopIteration:
                return
    
    return gen()

for batch in make_generator(reader.read_next_batch):
    writer.write_table(batch)

...but I wondered if something like this was possible with standard library functions or with some obscure syntax?

generator python
2 Answers

I would assume that the normal iter() function with its second argument should do what you want. As in:

for batch in iter(reader.read_next_batch, None):
    ...
user_7583450

The answer to your underlying question of how to iterate a CSVStreamingReader is: The CSVStreamingReader is iterable and does just the thing you want:

reader = pyarrow.csv.open_csv(...)
for batch in reader:
    ...

In general it is really rare for python libraries to return "iterable" things that are not python-iterable. That is always a sensible first thing to try.

user_7583450
Related