I trained a segmentation model in Pytorch and tested it to give an F1-score of 0.93 on my local computer (Windows, conda, CUDA 10.2, Pytorch 1.2). However, the F1-score dropped to 0.3 when testing on a Linux server (conda, CUDA 9.0, Pytorch 1.1). I double checked that both sets of code, label files, test sets were the same, and there is no "explicit" random sampling in my code (even if so, the effect shouldn't be so drastic).
I tried the following to solve my problem:
I suspected that it might have something to do with Pytorch versions, so installed Pytorch 1.1 on my Windows machine to match the server's, but got the same 0.93 score. I then thought it had to do with CUDA itself, so eliminated the GPU factor altogether and did inference on cpu on server, I still got 0.3 on server.
What could possibly be causing this huge discrepancy?