I have been using keras tensorboard callbacks and it was working pretty well. This week I wrote a new code and the callback is no longer storing scalars even when I'm specifying metrics in the model.compile() statement.
By default even without specifying metrics it used to grab the losses after every iteration in the earlier code.
Finally to check if there is a problem with tensorboard installation I tried running the old code again and it seems to be working fine but the new code doesn't. Only difference between the old and the new code is that I've used a datagenerator in the new code and used model.fit_generator() instead model.fit().
I tried reinstalling tensorboard but that didn't seem to solve the problem.
Old code -
def train(X,Y,val_split,checkpoint):
model=segmenter()
initial_epoch=0
if checkpoint is not None:
model.load_weights(checkpoint, by_name=True)
initial_epoch=int(re.search(r'\d+', os.path.basename(checkpoint)).group())
adam=Adam(lr=0.00001)
model.compile(optimizer=adam,loss='mean_squared_error')
filepath="checkpoint-{epoch:06d}.hdf5_50_50"
checkpoint = ModelCheckpoint(filepath, monitor='loss', verbose=1, save_best_only=True)
#for visualisation
visualisation= TensorBoard(log_dir='./logs', histogram_freq=0, batch_size=BATCH_SIZE, write_graph=True, write_grads=False, write_images=False, embeddings_freq=0, embeddings_layer_names=None, embeddings_metadata=None)
callbacks_list = [checkpoint,visualisation]
model.fit(x=X,y=Y,epochs=EPOCHS,batch_size=BATCH_SIZE, validation_split=val_split,callbacks=callbacks_list, shuffle=True, initial_epoch=initial_epoch )
New code -
def train(namelistfile,val_split=0.1,checkpoint=None,batch_size=1):
#getting file names
with open(namelistfile,'r') as f:
namelist=[line.replace('\n','') for line in f ]
# Generators
split=int((1-val_split)*len(namelist))
training_generator = mygenerator(namelist=namelist[:split], batch_size=batch_size)
validation_generator = mygenerator(namelist=namelist[split:], batch_size=batch_size)
# Training
model=autoencoder()
model.summary()
initial_epoch=0
if checkpoint is not None:
model.load_weights(checkpoint,by_name=True)
initial_epoch=int(re.search(r'\d+', os.path.basename(checkpoint)).group())
adam=Adam(lr=0.001)
model.compile(optimizer=adam,loss='mean_squared_error',metrics=['mean_squared_error'])
filepath="checkpoint-{epoch:06d}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='loss', verbose=1, save_best_only=True)
#for visualisation
visualisation= TensorBoard(log_dir='./logs', histogram_freq=0, batch_size=batch_size, write_graph=True, write_grads=False, write_images=False, embeddings_freq=0, embeddings_layer_names=None, embeddings_metadata=None)
callbacks_list = [checkpoint,visualisation]
model.fit_generator(generator=training_generator,validation_data=validation_generator,epochs=EPOCHS,callbacks=callbacks_list, initial_epoch= initial_epoch, use_multiprocessing=True,workers=6)
Following are the results to
tensorboard --inspect --logdir=logs
For old code
======================================================================
Processing event files... (this can take a few minutes)
======================================================================
Found event files in:
logs
These tags are in logs:
audio -
histograms -
images -
scalars
loss
val_loss
tensor -
======================================================================
Event statistics for logs:
audio -
graph
first_step 0
last_step 0
max_step 0
min_step 0
num_steps 1
outoforder_steps []
histograms -
images -
scalars
first_step 301
last_step 301
max_step 301
min_step 301
num_steps 1
outoforder_steps []
sessionlog:checkpoint -
sessionlog:start -
sessionlog:stop -
tensor -
======================================================================
For new code
======================================================================
Processing event files... (this can take a few minutes)
======================================================================
Found event files in:
logs
These tags are in logs:
audio -
histograms -
images -
scalars -
tensor -
======================================================================
Event statistics for logs:
audio -
graph
first_step 0
last_step 0
max_step 0
min_step 0
num_steps 1
outoforder_steps []
histograms -
images -
scalars -
sessionlog:checkpoint -
sessionlog:start -
sessionlog:stop -
tensor -
======================================================================
Tensorboard is giving no errors. It seems that I'm missing something and there is some problem with my code. Can anyone please help me with this as it being difficult to visualise without tensorboard.
TensorBoard 1.5.1
Keras 2.1.2
Tesorflow 1.4.1
Python 2.7.6