variable_scope does not get reused when using default scope name

user_9973879 0
Viewed 342

I have a question regarding sub-scopes when reusing variables. This

import tensorflow as tf

def make_bar():
  with tf.variable_scope('bar'):
    tf.get_variable('baz', ()) 

with tf.variable_scope('foo') as scope:
  make_bar()
  scope.reuse_variables()
  make_bar()

works perfectly fine, only a single variable foo/bar/baz is created.

However, if I change make_bar to

def make_bar(scope=None):
  with tf.variable_scope(scope, 'bar'):
    tf.get_variable('baz', ())

the code now fails with a 

ValueError: Variable foo/bar_1/baz does not exist

Question: why does variable scope reuse fail when using default names? If it is on purpose, what is the rationale behind this choice?

EDIT

Some precisions on the default_name argument of tf.variable_scope. From the documentation,

  • default_name: The default name to use if the name_or_scope argument is None, this name will be uniquified. If name_or_scope is provided it won't be used and therefore it is not required and can be None.

So as it name underlies, it is a way to provide a default scope name.

In the first version of make_bar, the scope name is forced to be bar -- the function has no parameter to change it.

In the second version of make_bar, I enhance this function to make it parameterizable. So bar is still the default scope name (provided this time as the default_name argument of tf.variable_scope), but this time the caller has the possibility to change it by setting the default argument scope of make_bar to anything else than None.`

When this second version of make_bar is used without an argument, it should, I think, fall back to the behavior of the first version -- which it does not.

Note that in my example, bar is intended to be a subscope of foo. The variable to be reused is meant to be foo/bar/baz.

tensorflow python
2 Answers

You're not actually using the scope 'foo' in your example. You need to pass the parameter to tf.variable_scope('foo', 'bar') or tf.variable_scope(scope, 'bar'). you're calling the method make_bar without the parameter in either case, which means in your first example name_or_scope='bar', in the second example name_or_scope=scope (with value None) and default_name='bar'.

this is probably what you want:

import tensorflow as tf

def make_bar(scope=None):
    with tf.variable_scope(scope, 'bar'):
        tf.get_variable('baz', ())

with tf.variable_scope('foo') as scope:
    make_bar(scope)
    scope.reuse_variables()
    make_bar(scope)

I would actually advise against using default parameters because they reduce readability like in your example. When is a None scope ever the answer you want? it would make more sense if you test it, maybe like this?

import tensorflow as tf

def make_bar(scope=None):
    if scope is None:
        scope = 'default_scope'
    with tf.variable_scope(scope, 'bar'):
        tf.get_variable('baz', ())


with tf.variable_scope('foo') as scope:
    make_bar(scope)   # use foo scope
    scope.reuse_variables()
    make_bar()        # use 'default_scope'

but this makes code less readable and more likely to lead to bugs

user_65654060

From the documentation https://www.tensorflow.org/api_docs/python/tf/variable_scope

If name_or_scope is not None, it is used as is. If name_or_scope is None, then default_name is used. In that case, if the same name has been previously used in the same scope, it will be made unique by appending _N to it.

...

To prevent accidental sharing of variables, we raise an exception when getting an existing variable in a non-reusing scope.

...

Similarly, we raise an exception when trying to get a variable that does not exist in reuse mode.

By using None scope you are forcing each call to get_variable to generate unique scope/variable, but by calling reuse_variables you are forcing subsequent calls to only return existing variables. There is no variable to return so ValueError is raised. This is expected behaviour.

user_65654060
Related