I'm basically working with grayscale images recorded with an ultrasound device. With these images a neural network (in my case LinkNet or U-Net with pretrained ResNet101) is trained and tested. Both networks are capable of processing grayscale and RGB images (different initialization for sure).
I want to know what the theoretical difference is when a neural network is working with grayscale instead of RGB images. The LinkNet was improving with grayscale images and the U-Net was deteriorating.
For sure there are slightly less model parameters to train but what is the impact of a one channel input on the neural network in terms of learning variance etc.? Or is there basically no real difference beside the different convolution dimensions?
Thank you!