Tensorflow Debugger v2 not showing any data

Question

I'm trying to debug a piece of code with tensorflow debugger v2, with the following instructions:

from network_definitions import *
import tensorflow as tf


os.environ["TF_CPP_MIN_LOG_LEVEL"] = "10"
os.environ["TF_DUMP_GRAPH_PREFIX"] = 'tbdump'
#os.environ["XLA_FLAGS"] = "--xla_dump_to=/tbdump/generated"
tf.debugging.set_log_device_placement(False)
tf.config.set_soft_device_placement(True)
tf.debugging.experimental.enable_dump_debug_info(
    './tbdump',
    tensor_debug_mode="FULL_HEALTH",
    circular_buffer_size=-1)


if __name__ == "__main__":

    big_dataset = 'gtzan'
    small_dataset = 'traintest_smallsmc'
    dataset = big_dataset
    # data_aug='NODAUG'# to run without data augmentation
    finetune_db(dataset, data_aug='DAUG', load_pkl=True)

but I'm running into the following issues:

after starting tensorboard --logdir /tbdump and accessing tensorboar on localhost, I always get the message "Debugger V2 is inactive because no data is available."
Sometimes, the execution of the program halts and never passes from "Epoch 1/50".

I can see that in tbdump folder there are being created the following type of files:

tfdbg_events.xxx...xxx.graphs
tfdbg_events.xxx...xxx.source_files
tfdbg_events.xxx...xxx.execution
tfdbg_events.xxx...xxx.stack_frames
tfdbg_events.xxx...xxx.graph_execution_traces
tfdbg_events.xxx...xxx.metadata

Any idea on how to make this work?

Tensorflow Debugger v2 not showing any data

0 Answers