How to pass values between steps of a scikit-learn pipeline?

Viewed 1706

I would like to use a pipeline, that uses a Vectorizer, followed by an LDA preprocessing step. The LDA preprocessing step needs the vocabulary_ of the Vectorizer.

How can I pass thus the vocabulary_ of the fitted Vectorizer step to the next LDA step? I tried to pass the pipeline itself to the LDA step, but this unfortunately does not work.

    pipe_full = Pipeline(
        [('vect', StemmedCountVectorizer(strip_accents='unicode', analyzer='word')),
         ('lda', SklLdaModel_mod()),
         ('clf', SGDClassifier(loss='log', penalty='elasticnet', n_iter=5, random_state=42, class_weight={0: 1, 1: 2}))])
    param_grid_full = [{'vect__ngram_range': ((1, 1), (1, 2)), 'vect__stop_words': (None, 'english'),
                        'vect__token_pattern': (r'(?u)\b\w\w+\b', r'(?u)\b([a-zA-Z]{3,})\b'),
                        'vect__stemmer': (None, SnowCastleStemmer(mode='NLTK_EXTENSIONS')),
                        'lda': (None, SklLdaModel_mod(id2word=pipe_full, num_topics=10), SklLdaModel_mod(id2word=pipe_full, num_topics=20)),
                        # 'lda__topics': (100, 200),
                        # 'lda__topics': (10, 20),  # for testing purposes only
                        'clf__alpha': (1e-4, 5e-4)}]

... and in the fit method of SklLdaModel_mod I have:

    if isinstance(self.id2word, Pipeline):
        try:
            self.id2word = {v: k for k, v in self.id2word.named_steps['vect'].vocabulary_.items()}

Any suggestions how to do this?

1 Answers
Related