Add file name and line number to tensorflow op name during debug mode

user_2930156 0
Viewed 103

I am interested in a feature or hacky solution that allows every tensorflow (specifically tf1.x) op name to include the file name and line number where the op is defined, in an automated fashion across the entire code base. This will greatly facilitate tracking down the place where an op raises an error, such as the situation below:

  File "tensorflow/contrib/distribute/python/mirrored_strategy.py", line 633, in _update
    assert isinstance(var, values.DistributedVariable), var
AssertionError: Tensor("floordiv_2:0", shape=(), dtype=int64, device=/job:chief/replica:0/task:0/device:GPU:0)

Right now the best I can do is to take a wild guess where a floordiv might occur, but honestly I have no clue at the moment.

tensorflow
1 Answers

The easiest way might be to show the graph on tensorboard and then search for failing op by name. Among the names of parent scopes or preceding operations you would probably be able to tell which layer is failing.

If not that, wrap your layer calls, model constructions in scopes. Hopefully, if this is your tensor failing and not a part of optimizer or else, you would see the direction where to look at.

If you are dedicated to wrap every op in a file-name/line-number, you can try to monkey-patch tf.Operation constructor with a scope. Which should be something along the next lines:

from inspect import getframeinfo, stack
import tensorflow as tf

def scopify_ops(func):
  def wrapper(*args, **kwargs):
    caller = getframeinfo(stack()[1][0])
    path = "%s:%d - %s" % (caller.filename, caller.lineno, message)
    print("Caller info:", path)
    with tf.name_scope("path")
      return func(*args, **kwargs)
  return wrapper

tf.Operation.__init__ = scopify_ops(tf.Operation.__init__)
user_8678890
Related