Custom Loss Function Leads to High MSE and an offset in the output Keras

I am training a neural network for time series regression. The model is

####################################################################################################################
# Define ANN Model
# define two sets of inputs
acc   =  layers.Input(shape=(3,1,))
gyro  =  layers.Input(shape=(3,1,))

# the first branch operates on the first input
x = Conv1D(256, 1, activation='relu')(acc)
x = Conv1D(128, 1, activation='relu')(x)
x = Conv1D(128, 1, activation='relu')(x)
x = MaxPooling1D(pool_size=3)(x)
x = Model(inputs=acc, outputs=x)

# the second branch opreates on the second input
y = Conv1D(256, 1, activation='relu')(gyro)
y = Conv1D(128, 1, activation='relu')(y)
y = Conv1D(128, 1, activation='relu')(y)
y = MaxPooling1D(pool_size=3)(y)
y = Model(inputs=gyro, outputs=y)


# combine the output of the three branches
combined =  layers.concatenate([x.output,  y.output])

# combined outputs
z = Bidirectional(LSTM(128, dropout=0.25, return_sequences=False,activation='tanh'))(combined)
z = Reshape((256,1),input_shape=(128,))
z = Bidirectional(LSTM(128, dropout=0.25, return_sequences=False,activation='tanh'))(combined)

#z = Dense(10, activation="relu")(z)
z = Flatten()(z)
z = Dense(4, activation="linear")(z)
model = Model(inputs=[x.input, y.input], outputs=z)
model.compile(loss=loss, optimizer = tf.keras.optimizers.Adam(),metrics=['mse'],run_eagerly=True)

I have tried to implement a custom loss function (based on different papers).

Math

The error will calculated as follows:

y_pred = [w x y z]
y_true = [w1 x1 y1 z1]
error = 2 * acos(w*w1 + x*x1 + y*y1 + z*z1)

Based on this formula I wrote the custom loss function:

def loss(y_true, y_pred):
    z = y_true * (y_pred )
    wtot = tf.reduce_sum(z,axis=1)
    error = 2*tf.math.acos(K.clip(tf.math.sqrt(wtot*wtot), -1.,1.))
    return error

But while the loss value is decreasing the MSE increased and I can see an offset in the output which will grow by the number of epochs. I understand that we do not optimize this Network for MSE but based on mathematics the MSE must be reduced or converge to some value near 1.

Orange is the Target/Reference

Blue is the Network ouptut

for 1 epoch

for 10 epochs

for 50 epochs