What is _uses_learning_phase in Keras?

user_3399944 0
Viewed 2040

I'm trying to write my own recurrent layer in Keras and noticed this line in the Keras source:

    # Properly set learning phase on output tensor.
    if 0 < self.dropout + self.recurrent_dropout:
        if training is None:
            output._uses_learning_phase = True

Checking the backend code for in_train_phase:

if training is None:
    training = learning_phase()
    uses_learning_phase = True
else:
    uses_learning_phase = False

This is rather confusing. Isn't "training" the "learning phase"?! I guess more importantly, do I need to set _uses_learning_phase on output in my custom recurrent layer?

keras tensorflow
1 Answers

Intro A "Training Flag" is meant to enable a Model (or Layer) to behave different from training when it predicts results or is being tested. Depending on the backend used, Keras may need to implement its own boolean "training flag" (on CNTK as for Keras 2.2.4) or can use a native backend tensor (like with Tensorflow) Therefor dynamic-purpose code was integrated.

As consequence Layer class has a property described as followed:

uses_learning_phase: Whether any operation
    of the layer uses `K.in_training_phase()`
    or `K.in_test_phase()`.

and output tensors may be given an attribute _uses_learning_phase which is read by the property. If any output tensor has the attribute (and it is true), the layer's property returns true.

Usage in Keras's Recurrent layer Your code snippet comes from keras/layers/recurrent.py and when calling the private _generate_dropout_mask method, the backend's operation creator "in_train_phase()" is being called. Therefore the output tensor's flag "_uses_learning_phase" is being set.

Explanation of quoted backend code

in_training_phase() and in_test_phase() are just the same. "training" is an optional argument and references the Training Flag. If the argument is not given, the Training Flag is refered automatically at

training = learning_phase()

However, the output tensor's attribute _uses_learning_phase is only set (and set True), if Training Flag is a tensor of the backend AND optional training argument was not set. (This may also explain, why a layer needs to set _uses_learning_phase itself, but I see no usecase for creating an operation via in_test_phase without flagging the output tensor. For now, assume there is one.)

user_108675280
Related