Layer.call is not executed eagerly

user_15026098 0
Viewed 140

I wrote a layer, which does nothing

class Fractal2D(tf.keras.layers.Layer):
    def __init__(self, kernel_size_range):
        super(Fractal2D, self).__init__()
        self.kernel_size_range = kernel_size_range
    
    def build(self, inputs):
        print(f'build executes eagerly: {tf.executing_eagerly()}')
        return inputs
    
    def call(self, inputs):
        print(f'call executes eagerly: {tf.executing_eagerly()}')
        return inputs

and made a model

model = tf.keras.Sequential([
    tf.keras.layers.InputLayer(input_shape=(224, 224, 3), batch_size=32),
    Fractal2D(kernel_size_range=(3, 41)),
    hub.KerasLayer("https://tfhub.dev/google/tf2-preview/mobilenet_v2/feature_vector/4", output_shape=[1280],
                   trainable=False),
    tf.keras.layers.Dense(DIAGNOSIS_NUMBER, activation='softmax')
])

the output from the cell is

build executes eagerly: True 
call executes eagerly: False

When I train the model

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
model.fit(training_set, validation_data=validation_set, epochs=20)

I get

 Epoch 1/20
 call executes eagerly: False
 call executes eagerly: False

Questions:

  1. Why build and call method are executed, when the model is instantiated?
  2. Why call method is NOT executed eagerly, if the eager execution is the default method of execution?
keras tensorflow deep-learning machine-learning python
2 Answers

The call method of the custom layer is automatically decorated with @tf.function, which essentially creates a dataflow graph on the first call and then executes this graph on all subsequent calls. Why is this relevant to your question? Because according to the docs on tf.executing_eagerly():

Eager execution is enabled by default and this API returns True in most of cases. However, this API might return False in the following use cases.

  • Executing inside tf.function, unless under tf.init_scope or tf.config.run_functions_eagerly(True) is previously called.
  • Executing inside a transformation function for tf.dataset.
  • tf.compat.v1.disable_eager_execution() is called.

So let's try see what happens when using tf.init_scope:

import tensorflow_hub as hub
import tensorflow as tf

class Fractal2D(tf.keras.layers.Layer):
    def __init__(self, kernel_size_range):
        super(Fractal2D, self).__init__()
        self.kernel_size_range = kernel_size_range
    
    def build(self, inputs):
        print(f'build executes eagerly: {tf.executing_eagerly()}')
        return inputs
    
    def call(self, inputs):
        with tf.init_scope():
          print(f'call executes eagerly: {tf.executing_eagerly()}')
        return inputs

model = tf.keras.Sequential([
    tf.keras.layers.InputLayer(input_shape=(224, 224, 3), batch_size=1),
    Fractal2D(kernel_size_range=(3, 41)),
    hub.KerasLayer("https://tfhub.dev/google/tf2-preview/mobilenet_v2/feature_vector/4", output_shape=[1280],
                   trainable=False),
    tf.keras.layers.Dense(1, activation='sigmoid')
])
training_set = tf.random.normal((1, 224, 224, 3))
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.fit(training_set, tf.random.normal((1, 1)), epochs=2)

build executes eagerly: True
call executes eagerly: True
Epoch 1/2
call executes eagerly: True
call executes eagerly: True
1/1 [==============================] - 4s 4s/step - loss: 0.2856 - accuracy: 0.0000e+00
Epoch 2/2
1/1 [==============================] - 0s 36ms/step - loss: 0.1641 - accuracy: 0.0000e+00
<keras.callbacks.History at 0x7f8836515710>

Seems to be consistent with the docs.

user_96578610
user_52355280
Related