Python/numpy/tensorflow execution time of function increases over time

UPDATE:

I managed to track the problem down to tensorflow.python._pywrap_tfe.TFE_Py_FastPathExecute The runtime of this function increases with each iteration:

Iteration 5: ncalls: 14269 total_time: 4.646 0.000 4.646 0.000 {built-in method tensorflow.python._pywrap_tfe.TFE_Py_FastPathExecute}

Iteration 19: ncalls: 16124 total_time: 6.594 0.000 6.594 0.000 {built-in method tensorflow.python._pywrap_tfe.TFE_Py_FastPathExecute}

ORIGINAL QUESTION:

I am currently working on an DQN. One iteration consists of sampling from the environment, adding the new data to the replay buffer and after that performing the training. The following function is used during sampling to normalize the images from the environment. The problem is that, the more iterations are performed, the slower this function executes.

from skimage.color import rgb2gray
from skimage.transform import resize

def preprocess_image(image, imgx, imgy):

    image = rgb2gray(image)
    image = np.expand_dims(image, axis = -1)
    image = resize(image,(imgx,imgy))
    image = (image-127.5)/127.5
    image = np.array(image, dtype = np.float32)

    return image

This function is called in here:

def sample_from_env(env : gym.Env, action : int, imgx : int, imgy : int, stack_frames : int, noops : int):

    observation, reward, done,_trunc, _ = env.step(action)

    observation = preprocess_image(observation,imgx, imgy)

    for i in range(stack_frames-1):

        loop_beginning = time.time()
    
        # if trajectory ended, end normal frame skipping
        if done == True:
            if i < stack_frames-1:
                new_observation = np.repeat(np.expand_dims(observation[:,:,-1],axis=-1),stack_frames-1-i,axis=-1)
                observation = np.concatenate([observation,new_observation],axis=-1)
            break

        new_observation, new_reward, done, _trunc, _  = env.step(noops)

        new_observation = preprocess_image(new_observation,imgx, imgy)

        observation = np.concatenate([observation,new_observation],axis=-1)
        reward += new_reward


    return observation,reward,done

And this function again is used in here:

def create_trajectory(model, epsilon : float,env_name : str ,frame_skips : int,imgx : int, imgy : int):
    s_a_r_s = []
    env = gym.make(env_name,full_action_space=False,new_step_api=True)

    # create action space
    ACTION_SPACE_SIZE = env.action_space.n # only discrete

    observation = env.reset()
    observation = preprocess_image(observation,imgx ,imgy)
    # bring start state into format of all other states
    observation = np.repeat(observation,frame_skips,axis=-1)

    for _ in range(10000):
   

        # epsilon greedy policy
        if rand.randint(0,100)<=epsilon*100:
            action = rand.randint(0,ACTION_SPACE_SIZE-1)
        else:
            with tf.device("/CPU:0"):
                action = int(tf.argmax(tf.squeeze(model(tf.expand_dims(observation,axis=0),training = False),axis = 0)).numpy())


        # sample from environment
        new_observation, reward, done = sample_from_env(env, action,imgx , imgy, stack_frames=frame_skips, noops = 0)


        s_a_r_s.append((observation,action,reward,new_observation, done))
        observation = new_observation

        if done:
            break

    return s_a_r_s

This is basically the code for one training iteration:

self.perform_sampling()
self.perform_training()

gc.collect()
tf.keras.backend.clear_session()

Some facts:

1. There is no gpu memory leak. In fact, the gpu is not even used in self.perform_sampling(). Every Tensorflow operation in here is done via with tf.device("/CPU:0").
2. The amount of ram usage for each training step does not increase and is always at roughly 49 percent
3. There is no memory leak (maybe only a small one of 2.48 mb)
4. The average time this functions takes to execute goes from 0.0008 to 0.004 seconds within 10 iterations. This means that one iteration goes from 30 seconds to 80 seconds!!. And this in a short amount of time
5. The time increase also happens when I use tf.image operations instead of sklearn and even if tf.image is performed on the GPU

I literally see no reason for this slow down and maybe one of you has seen something likes this before and knows were the reason is. Any help is welcome