LSTM and GRU vs SimpleRNN: "Type inference failed."

I've created a pretty simple sequential model, but my data is a inconvenient (each sample is a sequence of different lengths). That's OK, as each data item is relatively significant, so it works well to train with each sequence as a unique batch. Got that all working.

The model looks like:

Input(shape=(None, 42*3)) # I have a very preliminary dataset of 8 sequences of ~5000 frames holding 42 x/y/z floats.
Dense(256)
Dense(256)
SimpleRNN(61, return_sequences=True)

That's the whole thing. When I train for 100 epochs everything goes smoothly, maybe 45 seconds per epoch on my GTX 980ti.

When I try swapping out the SimpleRNN for a GRU or LSTM, however - which should be drop-in replacements in this context (if this is wrong, PLEASE correct me!), I start getting a weird error:

2022-07-27 21:18:15.989066: W tensorflow/core/common_runtime/forward_type_inference.cc:231] Type inference failed. This indicates an invalid graph that escaped type checking. Error message: INVALID_ARGUMENT: expected compatible input types, but input 1:
type_id: TFT_OPTIONAL
args {
  type_id: TFT_PRODUCT
  args {
    type_id: TFT_TENSOR
    args {
      type_id: TFT_LEGACY_VARIANT
    }
  }
}
 is neither a subtype nor a supertype of the combined inputs preceding it:
type_id: TFT_OPTIONAL
args {
  type_id: TFT_PRODUCT
  args {
    type_id: TFT_TENSOR
    args {
      type_id: TFT_FLOAT
    }
  }
}

        while inferring type of node 'cond_40/output/_19'

Additionally, the training happens MUCH faster - roughly 4-5s for the first epoch, then 1s per epoch afterward. That speedup leads me to suspect "something is wrong here".

My question: Am I safe to ignore this error/warning? If not, what's wrong, and how do I resolve it?

Side question: Are GRUs/LSTMs really that much faster to train, or is something wonky going on? I DO see that for the GRU and LSTM it's "Loaded cuDNN" which I think means it's CUDA-accelerated, but I don't see that anywhere for the SimpleRNN, so perhaps that's the difference?

EDIT: I was asked to include my data format, so here's the generator:

class MyBatchGenerator(keras.utils.Sequence):
    'Generates data for Keras'
    def __init__(self, shuffle=True):
        'Initialization'
        allDataPaths = list(sorted(glob.glob('PATH TO NPZ FILES SAVED EARLIER')))

        X = []
        Y = []
        for dp in allDataPaths:
            data = np.load(dp, allow_pickle=True)
            x = data['handData']
            x = x.reshape(x.shape[0], -1)
            y = np.array(data['keyData']).astype(float)
            y = y.reshape(y.shape[0], -1).astype(float)
            X.append(x)
            Y.append(y)

        maxLen = None
        self.X = tf.keras.preprocessing.sequence.pad_sequences(
            X, padding="post", value=-1.0, dtype='float', maxlen = maxLen
        )
        self.Y = tf.keras.preprocessing.sequence.pad_sequences(
            Y, padding="post", value=-1.0, dtype='float', maxlen = maxLen
        )

        self.shuffle = shuffle
        self.on_epoch_end()

    def __len__(self):
        'Denotes the number of batches per epoch'
        return len(self.Y)

    def __getitem__(self, index):
        return self.__data_generation(index)

    def on_epoch_end(self):
        'Shuffles indexes after each epoch'
        self.indexes = np.arange(len(self.Y))
        if self.shuffle == True:
            np.random.shuffle(self.indexes)

    def __data_generation(self, index):
        return self.X[index][np.newaxis], self.Y[index][np.newaxis]