When training a tf.keras (1.14) model, I keep seeing this warning:
Failed to find a tunable parameter that would decrease the output time. This means that the autotuning optimization got stuck in a local maximum. The optimization attempt will be aborted.
Training continues and go on for ever. Does anyone know how i can catch this warning and throw an error so I do not have the model training forever?
According to the script from Tensorflow in here, the hyparamers can not find better parameters during this epoch (by assuming the better param should decrease a bigger value than a delta value). This is just a warning that this may be the saturated optimal minima for the algorithm and no more increase in the training performance can take place.
You can ignore it and wait for upcoming epochs to truly verify the assumption because the code only considers the parameters in that epoch only. Sometimes, it takes more than an epoch to reach a better state.
