Build sklearn Pipeline with multiple inputs?

Viewed 149

I'm trying to work my way through this HackerRank challenge, where you are given three sets of input data:

  1. A paragraph of text (x_paragraph)
  2. A list of (5) questions (x_questions)
  3. A list of (5) answers to the questions in 2), but in a random order (x_answers)

The task is, given the paragraph of text, match the questions to the answers.

The approach I am trying to follow is first to extract features from the paragraph

pipeline = Pipeline([
    ('tfidf', TfidfVectorizer()),
    ('svd', TruncatedSVD(n_components=100))
])

pipeline.fit(x_paragraph)

Use this fitted pipeline to transform the questions and (unsorted) answers, and then find a measure of their similarity

xt_questions = pipeline.transform(x_questions)
xt_answers = pipeline.transform(x_answers)

similarity = cosine_similarity(xt_questions, xt_answers)

Finally, report the prediction as the most similar answer for each question

y_pred = [x_answers[np.argmax(similarities[i])] for i in range(len(x_answers))]

We are also provided with the correct answers (y) which we can use for training / validation.

My question is whether it is possible (and if so, how) to incorporate the additional input parameters (i.e. x_questions and x_answers) into the Pipeline so I can more easily evaluate the process and tune the parameters as necessary (i.e. incorporate GridSearchCV or such and make use of the true y)

I have seen this answer and have a vague feeling this should be possible with some combination of custom transformers, ColumnTransformer and / or FeatureUnion but I just can't quite seem to find the right approach whenever I've tried writing some code. Is there a viable recipe I can follow?

0 Answers
Related