TAPEX (large-sized model) cannot find average/difference

user_16656312 0
Viewed 21

I am using TAPEX for table question answering.

from transformers import AutoTokenizer, AutoModelForSeq2SeqL
import pandas as pd

tokenizer = AutoTokenizer.from_pretrained("microsoft/tapex-base-finetuned-wtq")
model = AutoModelForSeq2SeqLM.from_pretrained("microsoft/tapex-base-finetuned-wtq")

here is a table

data = {"Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"], "Number of movies": ["87", "53", "69"]}

I am trying to answer the following Questions: questions = ["how many movies does Leonardo Di Caprio have?","how many more movies does Leonardo Di Caprio have than George Clooney?","what's the first name of the actor who has 87 movies?"]

First and last are ok, but 2nd is incorrect. I have tried different question formats (1-4), even similar to the ones given on the website (Example). Test01, Test02, Test03, Test04

I have also tried

data['Number of movies'] = pd.to_numeric(data['Number of movies'])

because numbers were string. Still get 36.

The same problem with average, cannot find the mean/ave of number of movies. Test05 Test06

According to later exploration, TAPEX can use AVE operator, so what's the problem?

Is it because it is a "Language " AI & cannot do maths?

artificial-intelligence nlp
1 Answers

It's worth explaining something about complex language models like this: these models are very impressive in their ability to (apparently) "understand" a query and act appropriately most of the time. To a human, this can give the impression that they do understand and therefore can reliably perform certain tasks.

But what these models do is not understanding in any way, because Natural Language Understanding is not solved and may never be solved. These models are extremely advanced at reproducing the patterns that they extract from their training data, what they are good at is mimicking. Thus one can always unavoidably find a case where they fail, as long as the domain is infinite.

Of course, complex queries are more likely to derail the model: probably this model was trained only with simple queries about accessing a value, so it's simply not trained to calculate anything. An interesting question is why 36? My guess would be that it's just a value which appeared more often during training, so the model defaults to answering it when it has no clue what to do. It's worth noting that a better design would have been to train the model to recognize when it can't answer, and say "Sorry I don't know".

user_8919190
Related