when I run
python -m torch.distributed.run --nproc_per_node=8 --master_addr="127.0.0.1" --master_port=$RANDOM ~/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py --manual_loads_name l2l_resnet12rfs_cifarfs_adam_cl_80k
I get the error:
====> about to start train loop
Starting training!
Traceback (most recent call last):
File "/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/learn2learn/algorithms/maml.py", line 159, in adapt
gradients = grad(loss,
File "/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/autograd/__init__.py", line 226, in grad
return Variable._execution_engine.run_backward(
RuntimeError: CUDA out of memory. Tried to allocate 16.00 MiB (GPU 6; 39.59 GiB total capacity; 265.23 MiB already allocated; 10.19 MiB free; 282.00 MiB reserved in total by PyTorch)
learn2learn: Maybe try with allow_nograd=True and/or allow_unused=True ?
Traceback (most recent call last):
File "/home/miranda9/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py", line 216, in <module>
main()
File "/home/miranda9/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py", line 161, in main
train(args=args)
File "/home/miranda9/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py", line 196, in train
meta_train_iterations_ala_l2l(args, args.agent, args.opt, args.scheduler)
File "/home/miranda9/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/training/meta_training.py", line 149, in meta_train_iterations_ala_l2l
train_loss, train_loss_std, train_acc, train_acc_std = meta_learner(task_dataset, call_backward=True)
File "/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/miranda9/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/meta_learners/maml_meta_learner.py", line 371, in forward
meta_loss, meta_loss_std, meta_acc, meta_acc_std = forward(meta_learner=self,
File "/home/miranda9/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/meta_learners/maml_meta_learner.py", line 312, in forward
loss, acc = fast_adapt(
File "/home/miranda9/ultimate-utils/ultimate-utils-proj-src/uutils/torch_uu/meta_learners/maml_meta_learner.py", line 266, in fast_adapt
learner.adapt(adaptation_error)
File "/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/learn2learn/algorithms/maml.py", line 169, in adapt
self.module = maml_update(self.module, self.lr, gradients)
UnboundLocalError: local variable 'gradients' referenced before assignment
terminate called after throwing an instance of 'c10::CUDAError'
what(): CUDA error: unspecified launch failure
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Exception raised from query at ../aten/src/ATen/cuda/CUDAEvent.h:95 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f5036ae4a22 in /home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x132 (0x7f50dbec70e2 in /home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
frame #2: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x50 (0x7f50dbec8d40 in /home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
frame #3: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x11c (0x7f50dbec975c in /home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
frame #4: <unknown function> + 0xc71f (0x7f50da8aa71f in /home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0x7ea5 (0x7f50e43e6ea5 in /lib64/libpthread.so.0)
frame #6: clone + 0x6d (0x7f50e410fb0d in /lib64/libc.so.6)
/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 6 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 5 (pid: 158275) of binary: /home/miranda9/miniconda3/envs/meta_learning_a100/bin/python
/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py:367: UserWarning:
**********************************************************************
CHILD PROCESS FAILED WITH NO ERROR_FILE
**********************************************************************
CHILD PROCESS FAILED WITH NO ERROR_FILE
Child process 158275 (local_rank 5) FAILED (exitcode -6)
Error msg: Signal 6 (SIGABRT) received by PID 158275
Without writing an error file to <N/A>.
While this DOES NOT affect the correctness of your application,
no trace information about the error will be available for inspection.
Consider decorating your top level entrypoint function with
torch.distributed.elastic.multiprocessing.errors.record. Example:
from torch.distributed.elastic.multiprocessing.errors import record
@record
def trainer_main(args):
# do train
**********************************************************************
warnings.warn(_no_error_file_warning_msg(rank, failure))
Traceback (most recent call last):
File "/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/distributed/run.py", line 702, in <module>
main()
File "/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 361, in wrapper
return f(*args, **kwargs)
File "/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/distributed/run.py", line 698, in main
run(args)
File "/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/distributed/run.py", line 689, in run
elastic_launch(
File "/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 116, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 244, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
****************************************************************************************************************************************
/home/miranda9/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_dist_maml_l2l.py FAILED
========================================================================================================================================
Root Cause:
[0]:
time: 2022-02-04_20:26:43
rank: 5 (local_rank: 5)
exitcode: -6 (pid: 158275)
error_file: <N/A>
msg: "Signal 6 (SIGABRT) received by PID 158275"
========================================================================================================================================
Other Failures:
[1]:
time: 2022-02-04_20:26:43
rank: 6 (local_rank: 6)
exitcode: 1 (pid: 158280)
error_file: <N/A>
msg: "Process failed with exitcode 1"
****************************************************************************************************************************************
but different variations of the way to set up the script doesn't work.
In summary, how is one suppose to use torch.distributed.run such that TWO or MORE scripts can run at once and they are all distributed?
Related: