Are generators time efficent?

user_8266719 0
Viewed 70

I understand that generators in python atleast are memeory efficent as it deals with one item at a time but how does this make it time efficent (if it is) ?

Specifically, say I'm using generator function to load one data at a time for a machine learning task. At the end of the day, I will still need to loop over all the data elements and load them one at a time( using generator function). Yes, this is memeory efficent but this should instead take a lot more time to load the entire dataset than say loading all at once. Is my intuition right ?

#sample_code

def my_gen():
    for i in range(1000):
    features = np.random.randn(32,32,3)
    labels = np.random.randint(0,1, size = 1)
    yield features, labels
data-generation generator python
3 Answers

Treating a generator as a lazy sequence, it is usually less time efficient as a corresponding eager sequence.

%timeit sum((x*2 for x in range(5000)))  # lazy generator
366 µs ± 9.24 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit sum([x*2 for x in range(5000)])  # eager list
308 µs ± 3.12 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

This is because the generator holds intermediate state, which must be resumed for each item. In contrast, eagerly creating a sequence has to handle the intermediate state just once.

Keep in mind however that the overhead of generators is basically fixed. If each item takes a long time to compute, the constant overhead of the generator becomes negligible. When items are processed one-at-a-time, s also allow to free processed items, reducing the overall load on the process – possibly reaching a net time advantage at some point.

The advantage of generators is that lazyness allows to represent infinite sequences and latency – a generator is "n times O(i)" compared to a sequence "plain O(ni)". This allows a generator to produce each item at reliable time efficiency, even if the entire process would be delayed infinitely.

An infinite, eager sequence would have infinite time complexity but an infinite, lazy generator only produces items as needed.

def randoms():
    """Infinite stream of random numbers"""
    while True:
        yield random.random()

Likewise, generators allow external data sources time between providing each item. This can make a generator more efficient when the data source has notable latency between providing items.

user_53499160

No. Generators are, inherently, slower than similar alternatives (like list comprehensions).

But

If you're looking to reduce memory by loading data with generators, you probably shouldn't worry about this performance difference. It's more often the case that the performance bottleneck be at disk I/O and/or syscalls. The penalty by employing generators plays a negligible role in overall performance.

So the final answer is: Go ahead with generators. Its performance should be the last thing to worry about.

user_59584550

Generators are a feature you use. the implementation for a specific task is and whether it scales or not is a different story.
You can read a single item each time, and you can read a bulk of data each time and process them at each time. So depending on your situation, the later can be a much better option. in which case, it will be more time efficient as well.

user_27365590
Related