Quora Question Pairs challenge, predict if two questions ask the same thing using binary cross entropy loss to evaluate the predicition

user_10879926 0
Viewed 236

I have a csv file containing pairs of questions from the Quora Question Pairs Challenge. For each pair there is a corresponding label that specifies whether the questions are the same or not. I want to create a method so that if we have unknown pairs of questions I can answer if they ask the same thing or not. The accuracy of the result should be determined with the use of binary cross entropy loss.

This is a project that I have to do about a course of Information Retrieval. The problem is that all the solutions that I have found so far include Machine Learning (e.g. Neural Networks) and we haven't been taught how to use any Machine Learning models in this course. How can I solve this problem without using any Machine Learning?

I thought about cleaning the data (e.g. stop word reomval and punctuation removal) calculating the tf-idf and then applying cosine similarity between the two pairs. Like this I can find how similar two questions that are already given are, without using the labels. However, how can I use the labels to my advantage and predict the similarity between two unknown pairs of questions with no Machine Learning, is there a simple way that I am missing?

cosine-similarity tf-idf information-retrieval machine-learning nlp
1 Answers

You would require to use machine learning models to solve this problem. You have done the cleaning part that is nice and used the tf-idf to get the no of times the word has occurred in a given question. You can also try out word-2vec model which will also bring out the semantic meaning between the words. Infact quora uses random forest as its model to predict the similarity between two questions, You could check this link to get more detail. https://engineering.quora.com/Semantic-Question-Matching-with-Deep-Learning

Right now your solution is way too straightforward, although its good to start with. But i would suggest to get basic knowledge about models like logistic regression, decision tree to tackle this problem if you want better accuracy.

user_105063800
Related