Keras weight scaling- computational graph

Question

Model code:

vfnet = Sequential()
for (i, layeroutsize) in enumerate(hid_sizes):
    inshp = dict(input_shape=(ob_space.shape[0]+1,)) if i==0 else {} # add one extra feature for timestep
    vfnet.add(Dense(layeroutsize, activation=cfg["activation"], init="glorot_uniform",**inshp))
vfnet.add(Dense(1))

for layer in vfnet.layers:
    layer.W.set_value(layer.W.get_value(borrow=True)*wt_scale)

Question:

Keras version:1.0.1

Do lines 7-8 just happen once in the start or every time? I mean is it added to the computational graph, are the weights scaled by wt_scale every time?

Code to print graph:

import theano
for i in vfnet.layers:
    theano.printing.debugprint(i.input)
    theano.printing.debugprint(i.output)
ipdb.set_trace()

Graph(Confusing!!):

Using Theano backend.
[2017-07-22 17:07:27,139] Making new env: CartPole-v0
dense_input_2 [id A]
Elemwise{tanh,no_inplace} [id A] ''
 |Elemwise{add,no_inplace} [id B] ''
   |dot [id C] ''
   | |dense_input_2 [id D]
   | |dense_4_W [id E]
   |InplaceDimShuffle{x,0} [id F] ''
     |dense_4_b [id G]
Elemwise{tanh,no_inplace} [id A] ''
 |Elemwise{add,no_inplace} [id B] ''
   |dot [id C] ''
   | |dense_input_2 [id D]
   | |dense_4_W [id E]
   |InplaceDimShuffle{x,0} [id F] ''
     |dense_4_b [id G]
Elemwise{tanh,no_inplace} [id A] ''
 |Elemwise{add,no_inplace} [id B] ''
   |dot [id C] ''
   | |Elemwise{tanh,no_inplace} [id D] ''
   | | |Elemwise{add,no_inplace} [id E] ''
   | |   |dot [id F] ''
   | |   | |dense_input_2 [id G]
   | |   | |dense_4_W [id H]
   | |   |InplaceDimShuffle{x,0} [id I] ''
   | |     |dense_4_b [id J]
   | |dense_5_W [id K]
   |InplaceDimShuffle{x,0} [id L] ''
     |dense_5_b [id M]
Elemwise{tanh,no_inplace} [id A] ''
 |Elemwise{add,no_inplace} [id B] ''
   |dot [id C] ''
   | |Elemwise{tanh,no_inplace} [id D] ''
   | | |Elemwise{add,no_inplace} [id E] ''
   | |   |dot [id F] ''
   | |   | |dense_input_2 [id G]
   | |   | |dense_4_W [id H]
   | |   |InplaceDimShuffle{x,0} [id I] ''
   | |     |dense_4_b [id J]
   | |dense_5_W [id K]
   |InplaceDimShuffle{x,0} [id L] ''
     |dense_5_b [id M]
Elemwise{add,no_inplace} [id A] ''
 |dot [id B] ''
 | |Elemwise{tanh,no_inplace} [id C] ''
 | | |Elemwise{add,no_inplace} [id D] ''
 | |   |dot [id E] ''
 | |   | |Elemwise{tanh,no_inplace} [id F] ''
 | |   | | |Elemwise{add,no_inplace} [id G] ''
 | |   | |   |dot [id H] ''
 | |   | |   | |dense_input_2 [id I]
 | |   | |   | |dense_4_W [id J]
 | |   | |   |InplaceDimShuffle{x,0} [id K] ''
 | |   | |     |dense_4_b [id L]
 | |   | |dense_5_W [id M]
 | |   |InplaceDimShuffle{x,0} [id N] ''
 | |     |dense_5_b [id O]
 | |dense_6_W [id P]
 |InplaceDimShuffle{x,0} [id Q] ''
   |dense_6_b [id R]

Visualization:

from keras.utils.visualize_util import plot  
plot(vfnet, to_file='model.png')

Keras weight scaling- computational graph

1 Answers