Not fully connected layer in tensorflow

If you are looking for a solution for the specific example you provided, you can simply use tf.keras Functional API and define two Dense layers where one is connected to both neurons in the previous layer and the other one is only connected to one of the neurons:

from tensorflow.keras.layer import Input, Lambda, Dense, concatenate
from tensorflow.keras.models import Model

inp = Input(shape=(2,))
inp2 = Lambda(lambda x: x[:,1:2])(inp)   # get the second neuron 

h1_out = Dense(1, activation='sigmoid')(inp2)  # only connected to the second neuron
h2_out = Dense(1, activation='sigmoid')(inp)  # connected to both neurons
h_out = concatenate([h1_out, h2_out])

out = Dense(2, activation='sigmoid')(h_out)

model = Model(inp, out)

# simply train it using `fit`
model.fit(...)

The problem with your solution and some others suggested by other answers in this post is that they do not prevent training of this weight. They allow the gradient descent to train the non existent weight and then overwrite it retrospectively. This will result in a network that has a zero in this location as desired, but will negatively affect your training process as the back propagation calculation will not see the masking step as it is not part of a TensorFlow graph and so the gradient descent will follow a path which includes the assumption that this weight does have an affect on the outcome (it does not).

A better solution would be to include the masking step as a part of your TensorFlow graph, so that it can be factored into the gradient descent. Since the masking step is simply a element wise multiplication by your sparse, binary martix mask, you could just include the mask matrix as an elementwise matrix multiplicaiton in the graph definition using tf.multiply.

Sadly this means sying goodbye to the user friendly keras,layers methods and embracing a more nuts & bolts approach to TensorFlow. I can't see an obvious way to do it using the layers API.

See the implementation below, I have tried to provide comments explaining what is happening at each stage.

import tensorflow as tf

## Graph definition for model

# set up tf.placeholders for inputs x, and outputs y_
# these remain fixed during training and can have values fed to them during the session
with tf.name_scope("Placeholders"):
    x = tf.placeholder(tf.float32, shape=[None, 2], name="x")   # input layer
    y_ = tf.placeholder(tf.float32, shape=[None, 2], name="y_") # output layer

# set up tf.Variables for the weights at each layer from l1 to l3, and setup feeding of initial values
# also set up mask as a variable and set it to be un-trianable
with tf.name_scope("Variables"):
    w_l1_values = [[0, 0.25],[0.2,0.3]]
    w_l1 = tf.Variable(w_l1_values, name="w_l1")
    w_l2_values = [[0.4,0.5],[0.45, 0.55]]
    w_l2 = tf.Variable(w_l2_values, name="w_l2")

    mask_values = [[0., 1.], [1., 1.]]
    mask = tf.Variable(mask_values, trainable=False, name="mask")


# link each set of weights as matrix multiplications in the graph. Inlcude an elementwise multiplication by mask.
# Sequence takes us from inputs x to output final_out, which will be compared to labels fed to placeholder y_
l1_out = tf.nn.relu(tf.matmul(x, tf.multiply(w_l1, mask)), name="l1_out")
final_out = tf.nn.relu(tf.matmul(l1_out, w_l2), name="output")


## define loss function and training operation
with tf.name_scope("Loss"):
    # some loss defined as a function of graph output: final_out and labels: y_
    loss = tf.nn.sigmoid_cross_entropy_with_logits(logits=final_out, labels=y_, name="loss")

with tf.name_scope("Train"):
    # some optimisation strategy, arbitrary learning rate
    optimizer = tf.train.AdamOptimizer(learning_rate=0.001, name="optimizer_adam")
    train_op = optimizer.minimize(loss, name="train_op")


# create session, initialise variables and train according to inputs and corresponding labels
# This should show that the values of the first layer weights change, but the one set to 0 remains at 0
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    initial_l1_weights = sess.graph.get_tensor_by_name("Variables/w_l1:0")
    print(initial_l1_weights.eval())

    inputs = [[0.05, 0.10]]
    labels = [[0.01, 0.99]]
    ans = sess.run(train_op, feed_dict={"Placeholders/x:0": inputs, "Placeholders/y_:0": labels})

    train_steps = 1
    for i in range(train_steps):
        initial_l1_weights = sess.graph.get_tensor_by_name("Variables/w_l1:0")
    print(initial_l1_weights.eval())

Or use the answer provided by today for a keras friendly option.

You have multiple options here.

First, you could use the dynamic masking approach in your example. I believe this will work as expected since the gradients w.r.t. the masked-out parameters will be zero (the output is constant when you change the unused parameters). This approach is simple and it can be used even when your mask is not constant during the training.

Second, if you know beforehand which weights will be always zero, you can compose your weight matrix using tf.get_variable to get a submatrix, and then concatenate it with a tf.constant tensor, e.g.:

weights_sub = tf.get_variable("w", [dim_in, dim_out - 1])
zeros = tf.zeros([dim_in, 1])
weights = tf.concat([weights_sub, zeros], axis=1)

this example will make one column of your weight matrix to be always zero.

Finally, if your mask is more complex, you can use tf.get_variable on a flattened vector and then compose a tf.SparseTensor with the variable values on the used indices:

weights_used = tf.get_variable("w", [num_used_vars])
indices = ...  # get your indices in a 2-D matrix of shape [num_used_vars, 2]
dense_shape = tf.constant([dim_in, dim_out])  # this is the final shape of the weight matrix
weights = tf.SparseTensor(indices, weights_used, dense_shape)

EDIT: This probably won't work in combination with Keras' set_weights method, as it expects Numpy arrays, not Tensors.