How to define a prediction function in keras for NER system?

user_9808509 0
Viewed 178

I am creating a NER system following some tutorials using Keras. After the training and the first prediction, I'd like to use it to identify the NEs in a single string or in a list of strings of unseen data.

I can't seem to find the way to pass such string, or list of strings to the model.predict() and get an appropriate prediction.

This is the prediction for the test data in my code, so I was trying to adjust it to accept strings of unseen data and print the token + prediction:

i = np.random.randint(0, x_test.shape[0])
print("This is sentence:",i)
p = model.predict(np.array([x_test[i]]))
p = np.argmax(p, axis=-1)

print("{:15}{:5}\t {}\n".format("Word", "True", "Pred"))
print("-" *30)
for w, true, pred in zip(x_test[i], y_test[i], p[0]):
    print("{:15}{}\t{}".format(words[w-1], tags[true], tags[pred]))

This piece of code predicts and print each token with the NE Tag, but I don't really understand how it works

This code prints something like:

Word           True      Pred
------------------------------
The            O        O
British        B-gpe    B-gpe
pharmaceutical O        O
company        O        O
GlaxoSmithKlineB-org    O

I'd like to pass for example:

sentence = "President Obama became the first sitting American president to visit Hiroshima"

and being able to see the identified NEs. Any advise on how to do this?

A copy of the full code is here and the dataset is used is here.

named-entity-recognition keras tensorflow python
1 Answers

You can make a prediction on a list of sentences like this:

my_sentences = ["President Obama became the first sitting American president to visit Hiroshima",
                "Jack is a good person and living in Iran"]

my_sentences_idx = [[word2idx[w] for w in s.split(" ")] for s in my_sentences]

my_sentences_padded = pad_sequences(maxlen=max_len, sequences=my_sentences_idx, padding="post", value=num_words-1)
preds = np.argmax(model.predict(np.array(my_sentences_padded)), axis=-1)

for idx, p in enumerate(preds):
    print("-" *30)
    print(my_sentences[idx])
    print("-" *30)
    for w, pred in zip(my_sentences[idx].split(" "), preds[idx]):
        if tags[pred]!="O":
            print("{:15} {} ".format(w, tags[pred]))
    print()

Output:

------------------------------
President Obama became the first sitting American president to visit Hiroshima
------------------------------
President       B-per 
Obama           I-per 
American        B-gpe 
Hiroshima       B-geo 

------------------------------
Jack is a good person and living in Iran
------------------------------
Jack            B-per 
Iran            B-geo
user_24232780
Related