Keras - EarlyStopping based on user input

user_4841832 0
Viewed 1010

I am wondering if there is an easy way of creating a way of triggering early stopping in Keras based on user input rather than monitorization of any particular metric.

Ie I would like to send a keyboard signal to the process executing the training so that it gets out of the fit_generator function and execute the remaining code.

Any ideas?

EDIT: Based on @AnkurGoel 's answer, I wrote this code:

# Monitors the SIGINT (ctrl + C) to safely stop training when it is sent
flag = False
class TerminateOnFlag(Callback):
    """Callback that terminates training when the flag is raised.
    """
    def on_batch_end(self, batch, logs=None):
        if flag:    
            self.model.stop_training = True

def handler(signum, frame):
    logging.info('SIGINT signal received. Training will finish after this epoch')
    global flag
    flag = True

signal.signal(signal.SIGINT, handler) # We assign a specific handler for the SIGINT signal
terminateOnFlag = TerminateOnFlag()
callbacks.append(terminateOnFlag)

Where callbacks is a list of callbacks I fed into fit_generator.

During training, when I send the SIGINT signal indeed I get the message SIGINT signal received. Training will finish after this epoch, but when the epoch ends nothing happens. What is going on?

keras tensorflow python-3.x
3 Answers

You can give a thought to approach below:

Use One global variable, initialize 0 Use Signal Handler,

When signal(interrupt) received by the python process, its value is changed from 0 to 1.

Use Custom Callback in Keras, to stop the training when this variable value is changed

    class TerminateOnFlag(Callback):
    """Callback that terminates training when flag=1 is encountered.
    """

    def on_batch_end(self, batch, logs=None):
        if flag==1:    
            self.model.stop_training = True

Original Callbacks are available at: https://github.com/keras-team/keras/blob/master/keras/callbacks.py#L251

You still have to check if it is possible to provide custom callback to fit_generator, instead of standard callbacks.

Here is the code for signal Handler :

For windows:

    import signal, os

    def handler(signum, frame):
        print('Signal handler called with signal', signum)
        raise OSError("Couldn't open device!")

    signal.signal(signal.CTRL_C_EVENT, handler) # only in python version 3.2

For Linux:

    import signal, os

    def handler(signum, frame):
        print('Signal handler called with signal', signum)
        raise OSError("Couldn't open device!")

    signal.signal(signal.SIGINT, handler)
user_57375680

Better and safer way is to use mouse as input, for stopping, and other internal interactions.

For example, to stop keras in the end of batch when mouse is moved to the left side (mouse_x<10):

def queryMousePosition():
    from ctypes import windll, Structure, c_long, byref
    class POINT(Structure): _fields_ = [("x", c_long), ("y", c_long)]
    pt = POINT()
    windll.user32.GetCursorPos(byref(pt))
    return pt.x, pt.y  # %timeit queryMousePosition()


class TerminateOnFlag(keras.callbacks.Callback):
    def on_batch_end(self, batch, logs=None):
        mouse_x, mouse_y = queryMousePosition()
        if mouse_x < 10:
            self.model.stop_training = True

callbacks=[keras.callbacks.ReduceLROnPlateau(), TerminateOnFlag()]

model.fit_generator(..., callbacks=callbacks, ...)
user_28405310

Not using a keyboard signal, but when running Keras in a Jupyter notebook I found it easiest to use a callback that stops training on the presence of a particular file.

TRAINING_POISON_PILL_FILE_NAME = 'stop-training'

class PoisonPillCallback(tf.keras.callbacks.Callback): 
    def on_epoch_end(self, epoch, logs={}): 
        if os.path.exists(TRAINING_POISON_PILL_FILE_NAME):
            self.model.stop_training = True
            os.remove(TRAINING_POISON_PILL_FILE_NAME)
            print(f'poison pill file "{TRAINING_POISON_PILL_FILE_NAME}" detected, stopping training')

model.fit(..., callbacks=[PoisonPillCallback(), ...])

Then you can just creat an (empty) file with this name in the current directoy in the Jupyter UI and it will stop training after the current epoch.

user_3107310
Related