Q: Capture dimensions of transformer array for pipeline

user_1084896 0
Viewed 567

I'd like to create a sklearn pipeline using the keras sklearn wrapper. I am trying a sentiment classification task using the aclimdb, aka large movie dataset, which I have converted to a pandas dataframe of two columns, one for the review (string), and one for the label (integer).

> df.head(4)
                                              review  sentiment
0  "Lifeforce" is a truly bizarre adaptation of t...          1
1  I ordered this movie on the Internet as it is ...          0
2  he was my hero for all time until he went alon...          0
3  This is a 'sleeper'. It defines Nicholas Cage....          1

I have a pipeline that tokenizes the review using CountVectorizer, applies a tfidf transformation using TfidfTransformer, and then fits a binary classification model using KerasClassifier and the model function below:

X_train = df.loc[1:25000, "review"]
y_train = df.loc[1:25000, 'sentiment'].values
X_test = df.loc[25000:, "review"]
y_test = df.loc[25000:, 'sentiment'].values


np.random.seed(123) # for reproducibility

def model():
    model = models.Sequential([
        layers.Dense(16, input_shape = (10**4,), activation='relu'),
        layers.Dropout(0.5),
        layers.Dense(16, activation='relu'),
        layers.Dropout(0.5),
        layers.Dense(1, activation='sigmoid')
    ])
    model.compile(loss='binary_crossentropy', optimizer='rmsprop', 
                  metrics=['accuracy'])
    return model


early_stopping = callbacks.EarlyStopping(monitor='val_loss', patience=1, verbose=0, mode='auto')

pipe = pipeline.Pipeline([
    ('vect', CountVectorizer(max_features=10**4)),
    ('tfidf', TfidfTransformer()),
    ('nn', KerasClassifier(build_fn=model, 
                           nb_epoch=10, batch_size=128,
                           validation_split=0.2, callbacks=[early_stopping]))
])

In order to make this work, I had to specify the input_shape for the keras model, which means I have to fix the value of CountVectorizer's max_features. I'd like to not do this.

Is there a way I can obtain the dimensionality of the output from the previous pipeline stage, in this case, TfidfTransformer and pass that on to the KerasClassifier? i.e., something like this:

def model(input_df):
    model = models.Sequential([
        layers.Dense(16, input_shape = input_df.shape, activation='relu'),
        layers.Dropout(0.5),
        layers.Dense(16, activation='relu'),
        layers.Dropout(0.5),
        layers.Dense(1, activation='sigmoid')
    ])
    model.compile(loss='binary_crossentropy', optimizer='rmsprop', 
                  metrics=['accuracy'])
    return model
​
​
early_stopping = callbacks.EarlyStopping(monitor='val_loss', patience=1, verbose=0, mode='auto')
​
pipe = pipeline.Pipeline([
#    ('vect', CountVectorizer(max_features=10**4)),
#    ('tfidf', TfidfTransformer()),
    ('tfidf', TfidfVectorizer(max_features=10**4)),
    ('nn', KerasClassifier(build_fn=model(input_df=tfidf), 
                           nb_epoch=10, batch_size=128,
                           validation_split=0.2, callbacks=[early_stopping]))
])
​
## train network pipeline
​
pipe.fit(X_train.values, y_train)
​
-------------------------------------------------------------------
NameError                         Traceback (most recent call last)
<ipython-input-6-21be14eb185d> in <module>()
     19 #    ('tfidf', TfidfTransformer()),
     20     ('tfidf', TfidfVectorizer(max_features=10**4)),
---> 21     ('nn', KerasClassifier(build_fn=model(input_df=tfidf), 
     22                            nb_epoch=10, batch_size=128,
     23                            validation_split=0.2, callbacks=[early_stopping]))

NameError: name 'tfidf' is not defined

I could break the pipeline into two steps, and then save the output dataframe from the two transformers where I can easily capture the shape, but I'd rather do it in a single pass.

System information:

print(platform.platform())
print("Python", sys.version)
print("NumPy", np.__version__)
print("SciPy", scipy.__version__)
print("Scikit-Learn", sklearn.__version__)
print("Keras Backend", os.getenv("KERAS_BACKEND")) # doesn't work with tf https://github.com/fchollet/keras/issues/4984
​
Linux-4.4.0-91-generic-x86_64-with-debian-stretch-sid
Python 3.5.3 |Anaconda custom (64-bit)| (default, Mar  6 2017, 11:58:13) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
NumPy 1.13.3
SciPy 0.19.1
Scikit-Learn 0.19.0
Keras Backend cntk

Thanks!

scikit-learn keras machine-learning python
1 Answers

In order to solve this problem you have to:

  • remove input_shape from your model

  • define a custom ArrayTransformer for sklearnpipeline

  • insert this new transformer between tfidf/counter and keras model

In your code:

def model():
    model = models.Sequential([
       layers.Dense(16, activation='relu'),
       layers.Dropout(0.5),
       layers.Dense(16, activation='relu'),
       layers.Dropout(0.5),
       layers.Dense(1, activation='sigmoid')
    ])
    model.compile(loss='binary_crossentropy', optimizer='rmsprop', 
                  metrics=['accuracy'])
    return model


class ArrayTransformer():

   def transform(self, X, **transform_params):
       return X.toarray()

   def fit(self, X, y=None, **fit_params):
       return self


early_stopping = callbacks.EarlyStopping(monitor='val_loss', patience=1, verbose=0, 
mode='auto')

pipe = pipeline.Pipeline([
     ('tfidf', TfidfVectorizer(max_features=XXX)),
     ('transformer', ArrayTransformer()),
     ('nn', KerasClassifier(build_fn=model, 
                       nb_epoch=10, batch_size=128,
                       validation_split=0.2, callbacks=[early_stopping]))
   ])

pipe.fit(X_train.values, y_train)

In this way you can also combine tfidf/counter with GridSearchCV and tune min_df, max_features...

user_103750490
Related