How to remove connections between two nodes in TensorFlow model

I have an RGB image and my first layer is a Convolution layer with kernel size (1x1) and depth of 3 (filters). Because the filter is 1x1, the size of the output image doesn't change. Looking at this pixel vice, we see 3 nodes of the image connected to 3 nodes of the layer. When I try to access weights, I get a [ 3x3 ] matrix.

I want to remove certain connections between nodes and want only certain connections to update their weights during training.

As shown in the image above, I want to remove all the connections displayed by the dotted lines.

I want weights to look something like this.

[[ some_value NA         NA        ],
 [ NA         some_value NA        ],
 [ NA         NA         some_value]]

How can I achieve this in TensorFlow model?

Following is the model architecture I'm working with.

model = tf.keras.Sequential([
    keras.layers.Conv2D(3, (1,1), input_shape = (256, 256, 3), activation = "relu"),
    keras.layers.Conv2D(1, (1,1), activation = "relu"),
    ])