Text Image Combined Classification Model

user_6755253 0
Viewed 462

I am working on a problem statement where I have to match (text, image) pair. Given a furniture description and furniture image, I have to say they are same or not. This is a binary classification problem but I have to combine both text and image data. One possible solution I am trying as follows enter image description here In the above diagram, I am combining the feature from the pre-trained text and image model and training the linear layer end to end.

Is there any other way to handle this type of problem. any leads are most welcome. thanks a lot in advance for your help.

conv-neural-network deep-learning machine-learning nlp image-processing
1 Answers

I got a few recent works for the problem of "Image-Text Matching" which is slightly different from my problem statement but I can adjust the code for my project.

  1. Transformer Reasoning Network for Image-Text Matching and Retrieval enter link description here
  2. Stacked Cross Attention for Image-Text Matching enter link description here
user_67552530
Related