Can I reduce number of GPUs without terminating the training?

user_3943868 0
Viewed 132

Let's say I am using multiple GPUs (0,1,2,3) on one machine and later someone else also needs to use GPUs on this machine. Is there a way for me to reduce the number of gpu usage (i.e. only use 0 and 1) from my training without terminating the training and start over again? I don't want to waste the training I already did.

This sounds like a common need in a team. Is that possible?

huggingface-transformers pytorch tensorflow
1 Answers

I do not think that this is possible. You should save checkpoints so that you can later continue training where you left. This is possible with Hugging Face API.

training_args = Seq2SeqTrainingArguments(
        output_dir=model_directory,
        num_train_epochs=args.epochs,        
        do_eval=True,
        evaluation_strategy='epoch',
        load_best_model_at_end=True, # the last checkpoint is the best model wrt metric_for_best_model
        metric_for_best_model='eval_loss',
        greater_is_better=False, 
        save_total_limit=args.epochs   
    )

The save_total_limit is the number of checkpoints it will save. In the above case it would save a single checkpoint after each epoch. You can adjust the number based on your memory.

user_80105680
Related