I'm trying to work my way through this HackerRank challenge, where you are given three sets of input data:
- A paragraph of text (
x_paragraph) - A list of (5) questions (
x_questions) - A list of (5) answers to the questions in 2), but in a random order (
x_answers)
The task is, given the paragraph of text, match the questions to the answers.
The approach I am trying to follow is first to extract features from the paragraph
pipeline = Pipeline([
('tfidf', TfidfVectorizer()),
('svd', TruncatedSVD(n_components=100))
])
pipeline.fit(x_paragraph)
Use this fitted pipeline to transform the questions and (unsorted) answers, and then find a measure of their similarity
xt_questions = pipeline.transform(x_questions)
xt_answers = pipeline.transform(x_answers)
similarity = cosine_similarity(xt_questions, xt_answers)
Finally, report the prediction as the most similar answer for each question
y_pred = [x_answers[np.argmax(similarities[i])] for i in range(len(x_answers))]
We are also provided with the correct answers (y) which we can use for training / validation.
My question is whether it is possible (and if so, how) to incorporate the additional input parameters (i.e. x_questions and x_answers) into the Pipeline so I can more easily evaluate the process and tune the parameters as necessary (i.e. incorporate GridSearchCV or such and make use of the true y)
I have seen this answer and have a vague feeling this should be possible with some combination of custom transformers, ColumnTransformer and / or FeatureUnion but I just can't quite seem to find the right approach whenever I've tried writing some code. Is there a viable recipe I can follow?