Create a Neural Network in a For Loop with Tensorflow

Viewed 794

I am trying to create a simple MLP in a for loop with Tensorflow from a list containing the desired network structure e.g. structure = [100, 50, 20, 1]. In this list 100 represents the input size and 1 the output size. (I need this for a forecasting application but this is not strictly related to my question.)

I have not seen a similar way to create a network in Tensorflow around. For reasons that partially escape me people seem to suggest that it is better to declare each variable separately e.g. layer_1 = x1 * w1 + b1 then layer_2 = x2 * w2 + b2. Is the dynamic way to create the network I use in the for loop [for i in range(len(structure)-1):] wrong? To me the network seems to work fine and the network structure shown on tensorboard seems to be correct.

Do you think this way to create the network is fine? Do you think there are any Tensorflow / Context Manager issues I am falling in without even knowing it?

The network graph

Link to network graph

import tensorflow as tf

class Model(object):

    def __init__(self, structure, lr=0.01):

        assert structure[-1] == 1

        input_size = structure[0]

        act_fun = tf.nn.tanh

        G = tf.Graph()

        with G.as_default():

            self.X = tf.placeholder(tf.float32, shape=[None, input_size])
            self.Y = tf.placeholder(tf.float32, shape=[None, 1])              

            X_out = self.X
            for i in range(len(structure)-1):
                from_, to = structure[i], structure[i+1]
                initializer = tf.variance_scaling_initializer()
                w = tf.Variable(initializer([from_, to]), dtype=tf.float32, name=f'W{i}')
                b = tf.Variable(tf.zeros(to), name=f'B{i}')

                if to != 1:
                    X_out = act_fun(tf.matmul(X_out, w) + b)
                else:
                    X_out = tf.matmul(X_out, w) + b

            self.forecast_layer = X_out
            self.loss = tf.losses.mean_squared_error(self.Y, self.forecast_layer)
            self.trainer = tf.train.GradientDescentOptimizer(lr).minimize(self.loss)
            self.init = tf.global_variables_initializer()

        self.session = tf.Session(graph=G)
        self.session.run(self.init)
        self.session.graph.finalize()



    def fit(self, X, Y):
        self.session.run(self.trainer, feed_dict={self.X:X, self.Y:Y})



    def forecast(self, X):
        return self.forecast_layer.eval(feed_dict={self.X:X}, session=self.session)



    def evaluate_loss(self, X, Y):
        return self.loss.eval(feed_dict={self.Y:Y, self.forecast_layer:self.forecast(X)}, session=self.session)


M = Model([100, 50, 20, 1], lr=0.001)

1 Answers

This seems fine though I would use tf.layers.dense:

X_out = self.X
layers = [50, 20, 1]
for layer_size in layers:
    X_out = tf.layers.dense(X_out, layer_size, activation=act_fun if x != 1 else None)

Note:

Warning: THIS FUNCTION IS DEPRECATED. It will be removed in a future version. Instructions for updating: Use keras.layers.dense instead.

Related