Am I building this recurrent neural network correctly

So I'm attempting to build a neural network in Keras to classify a two variable input of audio data into two separate classes. The problem I'm having is I can't seem to get a validation accuracy above 80%, when I easily got 90+ with a variety of much simpler models. Also, when training, the accuracy starts at 20%, which seems weird to me since I'm doing binary classification.

Here's my model architecture:

snrLayers = LSTM(64, return_sequences=False, dropout=0.1, recurrent_dropout=0.1)(snr_lstm)
sigLayers = LSTM(64, return_sequences=False, dropout=0.1, recurrent_dropout=0.1)(sig_lstm)

dense_1 = Dense(64, activation='relu')(snrLayers)
dense_2 = Dense(64, activation='relu')(sigLayers)

dp_1 = Dropout(0.1)(dense_1, training=True)
dp_2 = Dropout(0.1)(dense_2, training=True)

output = Concatenate()(
    [
        dp_1,
        dp_2
    ]
)

output = Dense(1, activation='relu', name='weightedAverage_output')(output)


model = Model(
    inputs=[snr_lstm,sig_lstm],
    outputs=[output]
)

Yes, I've tried with more dense layers and and more dense parameters. No increase in performance.