Implementing a multi-input model in Keras, each with a different sample sizes each (different batch sizes each)

I am currently trying to implement a multi-input model in Keras. The input consists of multiple batches, and each includes different samples, but I get a 'different samples'-error. My implementation looks like this:

The model site looks as follows:

for s in range(NUM_STREAMS):
    inp.append(Input(shape=(16,8)))
...

The site where the error occurs:

history = model.train_on_batch(
                x=[x for x in X_batch],
                y=[y for y in y_batch]
            )

The error I get is:

ValueError: All input arrays (x) should have the same number of
samples. Got array shapes: [(6, 16, 8), (7, 16, 8), (6, 16, 8), (6, 16, 8)]

The abstract model architecture looks as follow: