Neural Network to count elements in text

Problem

I'm trying to create a neural network in keras(though i'm open to other suggestions) that would take the text (part of doctor's examination of a patient) and count the number of systems that were examined

Data Example

General: Alert and oriented, wel nourished.

Eye: EOMI, normal conjunctiva.

HENT: Normocephalic, normal hearing, moist oral mucosa, no scleral icterus,

Neck: Supple, non-tender, no lymphadenopathy.

Breast Exam: Not clinically necessary

Musculoskeletal: Norma range of motion and strength X3 bilateral upper extremity, left lower extremity, decreased range of motion

Skin: Skin is warm, dry and pink, no rashes or lesions, no ecchymosis.

Neurologic: Awake, alert, and oriented X3, no focal deficits.

Psychiatric: Cooperative, appropriate mood and affect.

The correct result for this example is 8 (by the count of systems/organs i've marked in bold) , excluding Breast Exam which was mentioned but not done

Training data and results

I have 50k labeled data points (the text + system counts) The data was cleaned of items that do not contribute to counting systems and was lemmatized so i'd say the quality of inputs is rather good.

So far the best results i got is around 40% accuracy , which is not good enough (ideally, i'd like to get to 90+% accuracy).

Question

My question is twofold:

1. If overall i'm moving in right direction, what prevents the model from giving better results?

2. If there is a better way to approach such task, what should i look at?

Thank you

What i have tried

I have tried a few approaches to build the model:

1. LSTM model with regression neuron on top(code below) - it tends to favor one specific output given enough epochs to learn - i've studied the existing questions on that and tried the suggestions - did not help (like changing the batch size, using class_weights, lowering the LR etc - did not help)
2. Replaced LSTM with Flatten + Dense layer(s) . The resulting accuracy was never higher than 40% , the result was +-1 of the correct result
3. Tried to see it as a classification problem , so LSTM followed by softmax activation. Same results as in #1 - highly favors one class
4. Removed embeddings and just passed tokenized and padded texts to dense layers - did not help
5. Played with hyperparameters for all the configurations above: embedding dims, hidden unit counts, batch size, padding length etc - mostly did not help
6. I considered just having a dictionary of systems and finding their occurrences in a text. But it would not be that easy as the system names could be misspelled, abbreviated, replaced with a synonym (like 'Respiratory' = 'lungs' = 'chest' etc) and we need to count out those systems that are mentioned but not reviewed and the wording for such cases can vary

Observations

1. In an LSTM attempts , i monitored the validation loss between batches during the training and it seems like it is not gradually decreasing but rather jumps - like if the model is predicting a '4' for every sample and at some point decides to predict '9', which overall gives better validation score
2. Spatial dropout after the Embedding gave like +10% accuracy
3. Lowering the padding size helped with accuracy (maybe we get some sort of vanishing gradient problem when there is too many zeros in padded sequences)

Code

X_train, X_test_initial, y_train, y_test = train_test_split(training_data['text'], training_data['num_exam'], test_size=0.02,
                                                        random_state=42)

    tokenizer = Tokenizer(num_words=5000,lower=True)
    tokenizer.fit_on_texts(training_data['text'])

    vocab_size = len(tokenizer.word_index) + 1

    maxlen = 40
    X_train = pad_sequences(tokenizer.texts_to_sequences(X_train), padding='post', maxlen=maxlen)
    X_test = pad_sequences(tokenizer.texts_to_sequences(X_test_initial), padding='post', maxlen=maxlen)

    embedding_dim = 50
    hidden_size = 50
    model = Sequential()
    model.add(Embedding(vocab_size, embedding_dim, input_length=maxlen))
    model.add(SpatialDropout1D(0.5))
    model.add(LSTM(hidden_size, dropout=0.2, recurrent_dropout=0.2))
    model.add(Dense(1, activation='relu'))
    model.compile(optimizer='adam',
                  loss='mean_squared_error',
                  metrics=['mse'])
    model.summary()

    history = model.fit(X_train, y_train, shuffle=False,
                        epochs=20, batch_size=128, verbose=1,
                        validation_split=0.02)