Default MaxPoolingOp only supports NHWC on device type CPU error when using trained model ported from GPU

Viewed 1644

Using python, keras and tensorflow I developed and trained a model on a PC with GPU ran predictions etc. everything works fine.

I then took the model & prediction code over to a laptop with requirements.txt rebuilt the environment swapping gpu packages to cpu packages.

When I try to run the prediction code I get an error which I cannot fathom.

I was under the impression that tensorflow would use/not use GPU transparently so I am left wondering what else it could be.

    Traceback (most recent call last):
  File ".\metatrader.py", line 231, in <module>
    result = predict(ret[0] + filename)
  File ".\metatrader.py", line 104, in predict
    array = model.predict(x)
  File "C:\Users\antho\.conda\envs\tensorflow\lib\site-packages\tensorflow\python\keras\engine\training.py", line 130, in _method_wrapper
    return method(self, *args, **kwargs)
  File "C:\Users\antho\.conda\envs\tensorflow\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1599, in predict
    tmp_batch_outputs = predict_function(iterator)
  File "C:\Users\antho\.conda\envs\tensorflow\lib\site-packages\tensorflow\python\eager\def_function.py", line 780, in __call__
    result = self._call(*args, **kwds)
  File "C:\Users\antho\.conda\envs\tensorflow\lib\site-packages\tensorflow\python\eager\def_function.py", line 846, in _call
    return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds)  # pylint: disable=protected-access
  File "C:\Users\antho\.conda\envs\tensorflow\lib\site-packages\tensorflow\python\eager\function.py", line 1843, in _filtered_call
    return self._call_flat(
  File "C:\Users\antho\.conda\envs\tensorflow\lib\site-packages\tensorflow\python\eager\function.py", line 1923, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "C:\Users\antho\.conda\envs\tensorflow\lib\site-packages\tensorflow\python\eager\function.py", line 545, in call
    outputs = execute.execute(
  File "C:\Users\antho\.conda\envs\tensorflow\lib\site-packages\tensorflow\python\eager\execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError:  Default MaxPoolingOp only supports NHWC on device type CPU
         [[node sequential/max_pooling2d_1/MaxPool (defined at .\metatrader.py:104) ]] [Op:__inference_predict_function_445]

Function call stack:
predict_function

Any help would be appreciated!!

Updated:

version I am using: tensorflow==2.3.1

After some further fiddling about I found that the training also doesn't work on the CPU only system but works fine with the GPU system. I expect this is some incompatibility from previous versions which I've not quite grasped.

import os
import sys
from tensorflow.keras import optimizers
from tensorflow.keras.layers import Dropout, Flatten, Dense, Activation, BatchNormalization
import tensorflow.keras.layers as lyrs
from tensorflow.keras.models import Sequential
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.preprocessing import image
import numpy as np
import tensorflow as tf

epochs = 100

train_data_dir = './data/train/'
validation_data_dir = './data/validate/'

nb_train_samples = sum([len(files) for r, d, files in os.walk(train_data_dir)])
nb_validation_samples = sum([len(files) for r, d, files in os.walk(validation_data_dir)])

nb_filters1 = 32
nb_filters2 = 32
nb_filters3 = 64
conv1_size = 3
conv2_size = 2
conv3_size = 5
pool_size = 2
classes_num = 2
batch_size = 128
chanDim =3

model = Sequential()
model.add(lyrs.Conv2D(nb_filters1, (conv1_size, conv1_size), input_shape=(150, 150, 3), padding='same'))
model.add(Activation('relu'))
model.add(lyrs.MaxPooling2D(pool_size=(pool_size, pool_size)))
model.add(lyrs.Conv2D(nb_filters2, (conv2_size, conv2_size), padding="same"))
model.add(Activation('relu'))
model.add(lyrs.MaxPooling2D(pool_size=(pool_size, pool_size), data_format="channels_first"))
model.add(lyrs.Conv2D(nb_filters3, (conv3_size, conv3_size), padding='same'))
model.add(Activation('relu'))
model.add(lyrs.MaxPooling2D(pool_size=(pool_size, pool_size), data_format="channels_first"))
model.add(Flatten())
model.add(Dense(1024))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(classes_num, activation='softmax'))
model.summary()

model.compile(loss='categorical_crossentropy',
                      optimizer=optimizers.RMSprop(),
                      metrics=['accuracy'])

train_datagen = ImageDataGenerator(
    horizontal_flip=False)

test_datagen = ImageDataGenerator(
    horizontal_flip=False)

train_generator = train_datagen.flow_from_directory(
    train_data_dir,
    target_size=(150, 150),
    batch_size=batch_size,
    class_mode='categorical'
)

validation_generator = test_datagen.flow_from_directory(
    validation_data_dir,
    target_size=(150, 150),
    batch_size=batch_size,
    class_mode='categorical')
    
model.fit_generator(
    train_generator,
    steps_per_epoch=nb_train_samples//batch_size,
    epochs=epochs,
    shuffle=True,
    validation_data=validation_generator,
    validation_steps=nb_validation_samples//batch_size)

model.save('./my_model.hdf5', overwrite=True)

The error I get:

tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError:  Default MaxPoolingOp only supports NHWC on device type CPU
         [[node sequential/max_pooling2d_1/MaxPool (defined at .\sample-training.py:72) ]] [Op:__inference_train_function_1065]
1 Answers

Ok I eventually found the answer.

There appears to be some inconsistency somewhere in tensorfol0w 2.3.1 related to how the model is trained on a GPU and CPU-only systems.

A model created and saved by the GPU system will not work when predictions are tried on a CPU-only system.

While the above code works perfectly well on the GPU system with the same version of tensorflow it fails with the above error on a CPU system and any model that is created/saved by the GPU PC when used to predict on a CPU only system also fails.

Changing the following lines resolves the problem

model.add(lyrs.MaxPooling2D(pool_size=(pool_size, pool_size), data_format="channels_first"))

to:

model.add(lyrs.MaxPooling2D(pool_size=(pool_size, pool_size), data_format="channels_last"))

It isn't clear to me yet what this exactly does however it also almost 1/2ves the number of trainable elements.

As I'm a newbie in the this area anyone with enlightening words pls feel free to leave a comment or correct me if/where I'm wrong.

Related