I'm running neural net with custom dataset on multiple nodes (4 nodes with 8 gpu each), however, some gpu within a node is out of memory and model.fit raised ResourceExhaustedError but do not terminate for about one day. I also try to run on 1 node with 4 gpu. When oom happens, model.fit terminate in 5 minute as expected.
The question is, running with TensorFlow MultiWorkerMirroredStrategy on more nodes when OOM happens on some gpu will hang without termination?
Info:
- Managing tools on HPC: Slurm
- Deep learning package: TensorFlow 2.3
Error log: link
Any help is appreciated!