Output of recurrent network always comes to a loop

I am trying to build a simple RNN for audio generation. However, after training network might generate some new data for a bit but then it loops on some number/pattern. I am not sure what the problem is.

Here is my network:

class rnn_model(tf.keras.Model):
  def __init__(self, chunk_size, out_size, rnn_units):
    super().__init__(self)
    self.inputs = tf.keras.layers.Input((chunk_size, 1))
    self.gru = tf.keras.layers.LSTM(rnn_units,
                                   return_sequences=True,
                                   return_state=True,
                                   bias_initializer='glorot_uniform')
    self.dense = tf.keras.layers.Dense(out_size, bias_initializer='glorot_uniform')
  def call(self, inputs, states=None, return_state=False, training=False):
    x = inputs
    '''if states is None:
      states = self.gru.get_initial_state(x)
    x, states = self.gru(x, initial_state=states, training=training)'''
    x = self.dense(x, training=training)

    if return_state:
      return x, states
    else:
      return x

And the output looks like this: Output

I wanted that it would create diverse patterns rather than just the looping one.