After installing PyTorch as per the official command:
conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=10.2 -c pytorch, my cuDNN version shown in conda list is pytorch 1.7.1 py3.8_cuda10.2.89_cudnn7.6.5_0 pytorch whereas my system has cudnn8.5.0.
Does it have an affect on how we train models?
TLDR; Probably no, but depends on the difference between versions.
In reality upgrades (like what you have conda cudnn7.6.5_0 -> cudnn8.5.0 of the system) usually don't harm training because versions are backward compatible for a while. After a while, things get deprecated though (years probably), so you should try to not totally make this absurdly large, such that CUDA version uses operations that aren't implemented. You should be more interested in which minimal version is required from your GPU (like 3090's from Nvidia which requires CUDA 11.1 and above).
In reality, torch uses simple operation implementation of cudnn, and usually, they don't change that much. As Nvidia describes what cuDNN is:
The NVIDIA CUDA® Deep Neural Network library (cuDNN) is a GPU-accelerated library of primitives for deep neural networks. cuDNN provides highly tuned implementations for standard routines such as forward and backward convolution, pooling, normalization, and activation layers.
If you are interested to check the actual matrix support from Nvidia, check this website documentation and click on the version support matrix of a specific version. You can see that cuDNN 8.5.0 for CUDA 10.2 is supported by 8.5.0. If you think what py3.8_cuda10.2.89_cudnn7.6.5_0 means, it is saying that your cuda10.2.89 was compiled with the primitives available in cudnn7.6.5.
Also, consider that you also have to have the required version of Nvidia driver.
