Extracting Data Logically from Unstructured Text

user_13601585 0
Viewed 47

Best method for parsing unstructured texts like below using NLP? Would a model be the best solution to this? If so, what would be the best way to start?

Prerequisite: Mathematics 408C and 408D with a grade of a C, and Physics 301 or 301K with a grade of a C.

which would result in something like

Mathematics 408C C, Mathematics 408D C, Physics 301 C or Physics 301 C

I've tried using just regex, but the sentence structure can be a lot more complex and inconsistent. Like below

Prerequisite: Architecture 415K with a grade of at least C; Mathematics 408C or 408K; and Physics 302K and 102M, or 303K and 103M.

Desired Result:

Architecture 415K C, Mathematics 408C or Mathematics 408K, Physics 302K and Physics 102M or Physics 303K and Physics 103M
nlp
1 Answers

Your input texts are not really natural language (full sentences containing information) but rather a semi-structured and therefore difficult to handle for both rule-based and semantics-based approaches.

A semantic/neural-model based approach like using a pre-trained Question Answering model in huggingface (python/PyTorch) might be a bit ouf of the box here but it could help to give some structure which is almost independent from previous structure:

enter image description here

enter image description here

Benefit of this approach is that is basically independent of the input structure, see for example an input with full sentences or a bullet point list.

Since the model only returns valid output if the question is clearly answerable from the context you would have to use a rule-based approach to get the list of Courses mentioned in the prerequisites, and then use the first answer to create a valid question for the grade.

But I honestly wonder how many possible ways to write there are for such a list. Also how much effort you want to put into this if this is just a handful of texts. If you show some more examples we can discuss possible rule-based approaches.

user_181896220
Related