Why does an instance of the keras Sequence iterate forever?

user_3999494 0
Viewed 1590

Here's a link to [the source code for the keras Sequence class][1].

I have created what I believe to be the exact same object by doing the following:

  1. copying and pasting the same __init__, __getitem__, and __len__ methods from the InfiniteGenerator class to the KGen class.
  2. copying the iter method from the source code of the Sequence class the the InfiniteGenerator class.

My hypothesis is that these two generators should both create a finite list, but the Sequence object will continue iterating forever. Why?

class InfiniteGenerator(object):
    def __init__(self, alist):
        self.alist = alist

    def __getitem__(self, idx):
        return self.alist[idx]

    def __len__(self):
        return len(self.alist)

    def __iter__(self):
        for item in (self[i] for i in range(len(self))):
            yield item

from keras.utils import Sequence

class KGen(Sequence):
    def __init__(self, alist):
        self.alist = alist

    def __getitem__(self, idx):
        return self.alist[idx]

    def __len__(self):
        return len(self.alist)


if __name__ ==  '__main__':
    ig = InfiniteGenerator(list(range(4)))
    for item in ig:
        print(item)


    print('now trying second iterator')

    import time
    time.sleep(1)

    kg = KGen(list(range(4)))
    for item in kg:
        print(item)
keras generator python
1 Answers

I ran into this as well using tensorflow.keras version 1.10. You can see in the source code they defined __iter__() to return an infinite generator. I added the following function to all of my Sequence classes to create a one-shot iterator for the situations where I need one.

def gen_iter(self):
    for i in range(len(self)):
        yield self[i]
user_16266940
Related