Problem when switching tensorflow data format from nhwc (channels last) to nchw (channels first)

user_4274552 0
Viewed 496

I am currently experimenting with training CRNN (convolutional recurrent neural networks) for an OCR task. I usually keep the data format as default, which is NHWC (channels last). But due to some deployment constraints, I want to use NCHW (channels first) data format.

To do this, I am using tensorflow.keras.backend.set_image_data_format('channels_first').

I am changing my data pipeline correspondingly by using:

if my_configs.data_format == 'channels_first':
    img = tf.transpose(img, [2, 0, 1])

My model has convolutional layers followed by bidirectionl lstm layers. This is part of the my model:

    conv_7 = Conv2D(64, (2,2), activation = "relu")(pool_6)
    
    if data_format == 'channels_first':
        squeezed = Lambda(lambda x: tf.squeeze(x, 2))(conv_7)
        # squeezed = Reshape((1,64,31))(squeezed)
    else:
        squeezed = Lambda(lambda x: tf.squeeze(x, 1))(conv_7)

    if data_format == "channels_first":
        use_bias = False
    else:
        use_bias = True
    # bidirectional LSTM layers with units=128
    blstm_1 = Bidirectional(LSTM(128, return_sequences=True, use_bias=use_bias))(squeezed)
    blstm_2 = Bidirectional(LSTM(128, return_sequences=True, use_bias=use_bias))(blstm_1)

Now, here's my problem. If I keep the bias in my LSTM layers, so use_bias=True, then I get an error regarding dimensions. This is the error:

Traceback (most recent call last):
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\framework\ops.py", line 1812, in _create_c_op
    c_op = pywrap_tf_session.TF_FinishOperation(op_desc)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Shape must be at least rank 3 but is rank 2 for '{{node BiasAdd}} = BiasAdd[T=DT_FLOAT, data_format="NCHW"](add, bias)' with input shapes: [?,512], [512].      

During handling of the above exception, another exception occurred:       

Traceback (most recent call last):   
  File "run_training.py", line 292, in <module>
    model = model_architecture_1_bis(len(char_list), data_format)
  File "D:\GdPicture\dev\MachineLearning\OCR\ModelsArchitectures.py", line 178, in model_architecture_1_bis    
    blstm_1 = Bidirectional(LSTM(128, return_sequences=True, use_bias=use_bias))(squeezed)
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\keras\layers\wrappers.py", line 530, in __call__
    return super(Bidirectional, self).__call__(inputs, **kwargs)
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 925, in __call__       
    return self._functional_construction_call(inputs, args, kwargs,       
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 1117, in _functional_construction_call
    outputs = call_fn(cast_inputs, *args, **kwargs)
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\keras\layers\wrappers.py", line 643, in call
    y = self.forward_layer(forward_inputs,
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\keras\layers\recurrent.py", line 663, in __call__        
    return super(RNN, self).__call__(inputs, **kwargs)
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 925, in __call__       
    return self._functional_construction_call(inputs, args, kwargs,       
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 1117, in _functional_construction_call
    outputs = call_fn(cast_inputs, *args, **kwargs)
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\keras\layers\recurrent_v2.py", line 1183, in call        
    runtime) = lstm_with_backend_selection(**normal_lstm_kwargs)
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\keras\layers\recurrent_v2.py", line 1557, in lstm_with_backend_selection
    last_output, outputs, new_h, new_c, runtime = defun_standard_lstm(    
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\eager\function.py", 
line 2828, in __call__
    graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\eager\function.py", 
line 3213, in _maybe_define_function 
    graph_function = self._create_graph_function(args, kwargs)
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\eager\function.py", 
line 3065, in _create_graph_function 
    func_graph_module.func_graph_from_py_func(
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\framework\func_graph.py", line 986, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\keras\layers\recurrent_v2.py", line 1305, in standard_lstm
    last_output, outputs, new_states 
= K.rnn(
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\util\dispatch.py", line 201, in wrapper
    return target(*args, **kwargs)   
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\keras\backend.py", line 4213, in rnn
    output_time_zero, _ = step_function(
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\keras\layers\recurrent_v2.py", line 1293, in step        
    z = K.bias_add(z, bias)
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\util\dispatch.py", line 201, in wrapper
    return target(*args, **kwargs)   
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\keras\backend.py", line 5772, in bias_add
    return nn.bias_add(x, bias, data_format='NCHW')
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\util\dispatch.py", line 201, in wrapper
    return target(*args, **kwargs)   
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 3366, in bias_add
    return gen_nn_ops.bias_add(
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\ops\gen_nn_ops.py", 
line 693, in bias_add
    _, _, _op, _outputs = _op_def_library._apply_op_helper(
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 742, in _apply_op_helper
    op = g._create_op_internal(op_type_name, inputs, dtypes=None,
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\framework\func_graph.py", line 591, in _create_op_internal
    return super(FuncGraph, self)._create_op_internal(  # pylint: disable=protected-access
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\framework\ops.py", line 3477, in _create_op_internal     
    ret = Operation(
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\framework\ops.py", line 1974, in __init__
    self._c_op = _create_c_op(self._graph, node_def, inputs,
  File "C:\Users\*\miniconda3\envs\tf2-gpu-env\lib\site-packages\tensorflow\python\framework\ops.py", line 1815, in _create_c_op
    raise ValueError(str(e))
ValueError: Shape must be at least rank 3 but is rank 2 for '{{node BiasAdd}} = BiasAdd[T=DT_FLOAT, data_format="NCHW"](add, bias)' with input shapes: [?,512], [512].

On the other hand, if I remove the bias, so use_bias=False, then I get no errors regarding dimensions but my model is terrible. I get around 50% accuracy (accuracy in this context means that I recognize a word correctly). I believe that these bad results are due to not using the bias in my LSTM layers but I could be wrong.

So my question is, can removing the bias affect the results that much? And also, what do you think is the source of that dimensions error when I use the bias?

recurrent-neural-network tensorflow2.0 tensorflow deep-learning ocr
2 Answers

So my question is, can removing the bias affect the results that much?

I doubt bias to be the reason for such low performance. If we take a step back to look at what's happening in the network, each layer is applying a series of linear transformation of the form W * X + B followed or not by a non linear transformation tanh, sigmoid....

The linear part W * X is the mathematical equivalent of applying several transformations such as : stretching space, contracting space, rotating....

Here is a pictures taken from the excellent 3 Blue 1 Brown channel

enter image description here

The X + B is the equivalent of shifting the origin in space, all else being equal.

From a Deep Learning perspective it would mean that a significant portion of the variance in the gradient lies along one direction in space and that by simply translating the input along that direction you decrease your cost function significantly (ie : compared to all the possible linear and non linear combination possible).

It's not impossible, but it seems like an unlikely event.

And also, what do you think is the source of that dimensions error when I use the bias?

See comments for more precision.

user_120545900

For a quick inspection you can train the model with channel last format with and without bias and monitor the accuracy, so if the bias have a major impact you can debug your channel first setup.

user_154364320
Related