Generator with memory leak

I have following batch generator:

def batch_generator(X, Y, array_size, batch_size = 96):
    while True:
            p = np.random.permutation(X.shape[0])
            X = X[p]
            Y = Y[p]
            for i in range(array_size):
                start = i*batch_size
                end = start + batch_size + 1
                yield X[start:end], Y[start:end]

And this section each while cycle use more and more memory.

            p = np.random.permutation(X.shape[0])
            X = X[p]
            Y = Y[p]

How to solve this leak?

Function is called by:

batch_gen = batch_generator(
    L[:(size_of_dataset*BATCH_SIZE + 1)],
    ab[:(size_of_dataset*BATCH_SIZE + 1)],
    size_of_dataset,
    BATCH_SIZE
)

And then keras fit

model.fit(batch_gen ,epochs=500...)

From this output, we can see, how memory usage growing:

MEM USED: 34028564480
Epoch 1/500
MEM USED: 68172357632
MEM USED: 68327460864
MEM USED: 94857654272
597/597 - 111s - accuracy: 0.6081 - loss: 0.0767 - val_accuracy: 0.6318 - val_loss: 0.0744
Epoch 2/500
MEM USED: 101374386176
MEM USED: 112654151680
597/597 - 103s - accuracy: 0.6365 - loss: 0.0721 - val_accuracy: 0.6416 - val_loss: 0.0716
Epoch 3/500
MEM USED: 102938615808
MEM USED: 112201596928
597/597 - 105s - accuracy: 0.6443 - loss: 0.0698 - val_accuracy: 0.6280 - val_loss: 0.0700
Epoch 4/500
MEM USED: 104500965376
MEM USED: 112002859008
597/597 - 104s - accuracy: 0.6502 - loss: 0.0683 - val_accuracy: 0.6612 - val_loss: 0.0680
Epoch 5/500
MEM USED: 106063552512
MEM USED: 111480320000
597/597 - 107s - accuracy: 0.6558 - loss: 0.0666 - val_accuracy: 0.6469 - val_loss: 0.0673
Epoch 6/500
MEM USED: 107625590784
MEM USED: 110876241920
597/597 - 105s - accuracy: 0.6626 - loss: 0.0651 - val_accuracy: 0.6636 - val_loss: 0.0661
Epoch 7/500