If I close my JupyterLab from notebook instance, would my code be gone?

user_14679731 0
Viewed 305

I'm new to AWS and I'm trying out AWS Sagemaker. I'm currently doing my project which involves quite a long time to finish and I don't think I can finish it in a day. I'm worried if I close my JupyterLab of my notebook instance in SageMaker, my code will be gone. How do I save my code and cell run progress when using Sagemaker?

amazon-sagemaker tensorflow amazon-web-services python
1 Answers

If you are training directly in the notebook the answer is yes. However the best practice is not to train directly with the notebook. Use instead the notebook (you can choose a very cheap instance for the notebook) to launch your training job (in the instance type you desire) adapting you code to be the entrypoint of the estimator. In that way, you can close the notebook after launching the training job and monitor the training job using cloudwatch. You can also define some regex to capture metrics from the stout and cloudwatch will automatically plot for you, which is very useful! As a quick example.. in my notebook I have this cell:

import sagemaker from sagemaker.tensorflow import TensorFlow from sagemaker import get_execution_role

bucket = 'mybucket'

train_data = 's3://{}/{}'.format(bucket,'train')

validation_data = 's3://{}/{}'.format(bucket,'test')

s3_output_location = 's3://{}'.format(bucket)

hyperparameters = {'epochs': 70, 'batch-size' : 32, 'learning-rate' :
0.01}

metrics = [{'Name': 'Loss', 'Regex': 'loss: ([0-9\.]+)'},
           {'Name': 'Accuracy', 'Regex': 'acc: ([0-9\.]+)'},
           {'Name': 'Epoch', 'Regex': 'Epoch ([0-9\.]+)'},
           {'Name': 'Validation_Acc', 'Regex': 'val_acc: ([0-9\.]+)'},
           {'Name': 'Validation_Loss', 'Regex': 'val_loss: ([0-9\.]+)'}]

tf_estimator = TensorFlow(entry_point='training.py', 
                          role=get_execution_role(),
                          train_instance_count=1, 
                          train_instance_type='ml.p2.xlarge',
                          train_max_run=172800,
                          output_path=s3_output_location,
                          framework_version='1.12',
                          py_version='py3',
                          metric_definitions = metrics,
                          hyperparameters = hyperparameters)

inputs = {'train': train_data, 'test': validation_data}

myJobName = 'myname'

tf_estimator.fit(inputs=inputs, job_name=myJobName)

My training script training.py is something like this:

if __name__ =='__main__':

    parser = argparse.ArgumentParser()

    # input data and model directories
    parser.add_argument('--gpu-count', type=int, default=os.environ['SM_NUM_GPUS'])
    parser.add_argument('--model-dir', type=str, default=os.environ.get('SM_MODEL_DIR'))
    parser.add_argument('--train', type=str, default=os.environ.get('SM_CHANNEL_TRAIN'))
    parser.add_argument('--test', type=str, default=os.environ.get('SM_CHANNEL_TEST'))
    parser.add_argument('--learning-rate', type=float, default=0.0001)
    parser.add_argument('--batch-size', type=int, default=32)
    parser.add_argument('--epochs', type=int, default=1)
....
user_42674390
Related