GPT3 : from next word to Sentiment analysis, Dialogs, Summary, Translation ....?

user_1019129 0
Viewed 412

How does GPT3 or other model goes from next word prediction to do Sentiment analysis, Dialogs, Summaries, Translation .... ?

what is the idea and algorithms ? How does it work ?

F.e. generating paragraph is generate next word then the next ..next..

On the other hand Sentiment analysis task is paragraph of text is Good/Bad, which is a classification ? Extracting meaningful sentence from paragraph is even more different task.

How do we go from next token to ...... !

Andre thanks for the replies.

It seems my question is not clear enough. So let me elaborate. Next-token prediction can be trained on normal text corpus.

word1 w2 w3 w4 .....

Next Sentiment can be trained on sentence=>marker=>label

sent1: word1 w2 w3 w4 ..... marker label1
sent2: word1 w2 w3 w4 ..... marker label2
sent3: word1 w2 w3 w4 ..... marker label3
....

It is no longer corpus-next-token-generation. It is next-token generation. The problem is you need to have the LABALED data !!

How about text summation ... lets use keyword extraction (and eventually sentence selection based on those keywords) Again u need even more complex labeling.

  paragraph1 => kw1
  paragraph1 => kw2
  paragraph2 => kw3
  paragraph3 => kw4

it still can be thought of as next-token prediction but you need again specialized LABELED data.

So my question given ONLY corpus text, how do you do the Sentiment, Text summary .... etc ?

Otherwise GPT3 is simply scaled DNN with thousands of man hours for labeling data !!

WHERE is the LEAP ?

gpt artificial-intelligence nlp
1 Answers

GPT-3 is a few-shot learning Natural Language Generator.

It's possible to adapt those generic models to perform specific tasks, such as classification, translation, summaries, etc.

For such, we need to:

  1. Fine-tune the model for the desired task
  2. Pre-process the query
  3. Query the NLG (call API)
  4. Post-process the answer

Example:

1- Fine tuning:

This is a tweet sentiment classifier


Tweet: "I loved the new Batman movie!"
Sentiment: Positive
###
Tweet: "I hate it when my phone battery dies."
Sentiment: Negative
###
Tweet: "My day has been "
Sentiment: Positive
###
Tweet: "This is the link to the article"
Sentiment: Neutral
###

2- Query sample:

This new music video blew my mind

2.1 - Pre-process query (encapsulate the query in the same format as found in the the fine tuning):

Tweet: "This new music video blew my mind"
Sentiment:

3 - Query the API. The expected result should be:

 Positive
###

4 - Post-processing / format:

  • Remove trailing spaces, '###' and any other extra line.
  • Try to exact-match result between ['Positive', 'Neutral', 'Negative']
  • If any other answer is found, handle the errors.

I hope this example is clarifying enough. You can find more information on the documentation and on examples.

user_37394100
Related