Keras Tensorboard callback not logging scalars(loss)

Viewed 643

I have been using keras tensorboard callbacks and it was working pretty well. This week I wrote a new code and the callback is no longer storing scalars even when I'm specifying metrics in the model.compile() statement.

By default even without specifying metrics it used to grab the losses after every iteration in the earlier code.

Finally to check if there is a problem with tensorboard installation I tried running the old code again and it seems to be working fine but the new code doesn't. Only difference between the old and the new code is that I've used a datagenerator in the new code and used model.fit_generator() instead model.fit().

I tried reinstalling tensorboard but that didn't seem to solve the problem.

Old code -

def train(X,Y,val_split,checkpoint):

    model=segmenter()
    initial_epoch=0
    if checkpoint is not None:
        model.load_weights(checkpoint, by_name=True)
        initial_epoch=int(re.search(r'\d+', os.path.basename(checkpoint)).group())

    adam=Adam(lr=0.00001)
    model.compile(optimizer=adam,loss='mean_squared_error')
    filepath="checkpoint-{epoch:06d}.hdf5_50_50"
    checkpoint = ModelCheckpoint(filepath, monitor='loss', verbose=1, save_best_only=True)
    #for visualisation
    visualisation= TensorBoard(log_dir='./logs', histogram_freq=0, batch_size=BATCH_SIZE, write_graph=True, write_grads=False, write_images=False, embeddings_freq=0, embeddings_layer_names=None, embeddings_metadata=None)
    callbacks_list = [checkpoint,visualisation]
    model.fit(x=X,y=Y,epochs=EPOCHS,batch_size=BATCH_SIZE, validation_split=val_split,callbacks=callbacks_list, shuffle=True, initial_epoch=initial_epoch )

New code -

def train(namelistfile,val_split=0.1,checkpoint=None,batch_size=1):

    #getting file names
    with open(namelistfile,'r') as f:
        namelist=[line.replace('\n','') for line in f ]

    # Generators
    split=int((1-val_split)*len(namelist))
    training_generator = mygenerator(namelist=namelist[:split], batch_size=batch_size)
    validation_generator = mygenerator(namelist=namelist[split:], batch_size=batch_size)

    # Training
    model=autoencoder()
    model.summary()
    initial_epoch=0
    if checkpoint is not None:
        model.load_weights(checkpoint,by_name=True)
        initial_epoch=int(re.search(r'\d+', os.path.basename(checkpoint)).group())
    adam=Adam(lr=0.001) 
    model.compile(optimizer=adam,loss='mean_squared_error',metrics=['mean_squared_error'])
    filepath="checkpoint-{epoch:06d}.hdf5"
    checkpoint = ModelCheckpoint(filepath, monitor='loss', verbose=1, save_best_only=True)
    #for visualisation
    visualisation= TensorBoard(log_dir='./logs', histogram_freq=0, batch_size=batch_size, write_graph=True, write_grads=False, write_images=False, embeddings_freq=0, embeddings_layer_names=None, embeddings_metadata=None)
    callbacks_list = [checkpoint,visualisation]
    model.fit_generator(generator=training_generator,validation_data=validation_generator,epochs=EPOCHS,callbacks=callbacks_list, initial_epoch= initial_epoch, use_multiprocessing=True,workers=6)

Following are the results to

tensorboard --inspect --logdir=logs

For old code

======================================================================
Processing event files... (this can take a few minutes)
======================================================================

Found event files in:
logs

These tags are in logs:
audio -
histograms -
images -
scalars
   loss
   val_loss
tensor -
======================================================================

Event statistics for logs:
audio -
graph
   first_step           0
   last_step            0
   max_step             0
   min_step             0
   num_steps            1
   outoforder_steps     []
histograms -
images -
scalars
   first_step           301
   last_step            301
   max_step             301
   min_step             301
   num_steps            1
   outoforder_steps     []
sessionlog:checkpoint -
sessionlog:start -
sessionlog:stop -
tensor -
======================================================================

For new code


======================================================================
Processing event files... (this can take a few minutes)
======================================================================

Found event files in:
logs

These tags are in logs:
audio -
histograms -
images -
scalars -
tensor -
======================================================================

Event statistics for logs:
audio -
graph
   first_step           0
   last_step            0
   max_step             0
   min_step             0
   num_steps            1
   outoforder_steps     []
histograms -
images -
scalars -
sessionlog:checkpoint -
sessionlog:start -
sessionlog:stop -
tensor -
======================================================================


Tensorboard is giving no errors. It seems that I'm missing something and there is some problem with my code. Can anyone please help me with this as it being difficult to visualise without tensorboard.

TensorBoard 1.5.1
Keras 2.1.2
Tesorflow 1.4.1
Python 2.7.6 
0 Answers
Related