I'm implementing a question answering assignment using Spark. There are two dataframes are involved in the assignment. Dataframes are called "articles" and "questions", each row of "articles" is one article, and each row of "questions" is represented as one question. The goal is to find the most relevant article for each question.
So, there's the problem. Suppose there are "m" rows for "articles" and "n" rows for "questions". I used cross join on two dataframes, so a new joined dataframe contained "m * n" rows. Then I applied the UDF function on each row and derived a similarity score for each question and article pair row. And pick out the top score row as the answer based on each question.
But the running time of this method isn't bearable when "m * n" is large enough. So, I'm thinking is there a better way to solve this problem?
The following is the code I wrote: https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/1135489547726938/676505825717012/1633342303836206/latest.html