Subnetwork weight sharing over different scopes in tensorflow

Viewed 262

Working with tensorflow, I am trying to share identical weights from identical networks in different variable scopes to save memory. However there seems to be no simple way to do this. I have prepared a small code sample to illustrate on a smaller scale what I would like to do with a larger subnetwork:

import tensorflow as tf

graph = tf.Graph()
with graph.as_default():
    with tf.variable_scope("super_scope_one"):
        scope1 = tf.variable_scope("sub_scope_one")
        with scope1:
            number_one = tf.get_variable("number_one", shape=[1],
                                         initializer=tf.ones_initializer)
    with tf.variable_scope("super_scope_two"):
        with tf.variable_scope("sub_scope_one", reuse=True) as scope2:
            # Here is the problem.
            # scope1.reuse_variables() # this crashes too if reuse=None.
            number_one = tf.get_variable("number_one", shape=[1])
        with tf.variable_scope("sub_scope_two"):
            number_two = tf.get_variable("number_two", shape=[1],
                                         initializer=tf.ones_initializer)
        number_three = number_one + number_two

    init_op = tf.global_variables_initializer()

with tf.Session(graph=graph):
    init_op.run()
    print(number_three.eval())

Is there a way to share the variables in the two sub scopes, without removing the scopes above? And if not is there a good reason why this would be a bad Idea?

1 Answers

You can simply define number_one only once in "super_scope_one" and use it in "super_scope_two" as well.

Two variables in different scopes can be used together. See below:

import tensorflow as tf

graph = tf.Graph()
with graph.as_default():
    with tf.variable_scope("super_scope_one"):
        scope1 = tf.variable_scope("sub_scope_one")
        with scope1:
            number_one = tf.get_variable("number_one", shape=[1],
                                         initializer=tf.ones_initializer)
    with tf.variable_scope("super_scope_two"):
        with tf.variable_scope("sub_scope_two"):
            number_two = tf.get_variable("number_two", shape=[1],
                                         initializer=tf.ones_initializer)
        number_three = number_one + number_two

    init_op = tf.global_variables_initializer()

    with tf.Session(graph=graph):
        init_op.run()
        print(number_three.eval())

Returns [2]

Related