SageMaker using Debugger in local mode

Viewed 436

Abstract

I'm trying to test SageMaker Debugger examples in 'local mode' from amazon-sagemaker-examples. I'm trying to see the same debugging result in 'local mode' with AWS SageMaker Notebook instance.

What i've done

I put a few lines in one of the examples to use it in 'local mode' referenced from amazon-sagemaker-local-mode-example. The example is tf-mnist-builtin-rule.ipynb. Commented line is original code in the example

import subprocess
# import boto3
from sagemaker.local import LocalSession
from sagemaker.tensorflow import TensorFlow

instance_type = 'local'
try:
    if subprocess.call("nvidia-smi") == 0:
        instance_type = "local_gpu"
except:
    pass
session = LocalSession()
session.config = {'local': {'local_code': True}}
# session = boto3.session.Session()
# region = session.region_name
role = 'arn:aws:iam::111111111111:role/service-role/AmazonSageMaker-ExecutionRole-20200101T000001'
estimator = TensorFlow(
    # role=sagemaker.get_execution_role(),
    role=role,
    instance_count=1,
    # instance_type="ml.p3.8xlarge",
    instance_type=instance_type,
    # image_uri=f"763104351884.dkr.ecr.{region}.amazonaws.com/tensorflow-training:2.3.1-gpu-py37-cu110-ubuntu18.04",
    framework_version='2.4.1',
    py_version="py37",
    max_run=3600,
    source_dir="./src",
    entry_point="tf-resnet50-cifar10.py",
    # Debugger Parameters
    rules=built_in_rules,
    sagemaker_session=session,
)

Problem:

I can't see debugging report in realtime unlike AWS SageMaker Notebook instance. The example said wait=false option (see the figure) makes notebook proceed though it is still running but not working in 'local mode'.

enter image description here

Question:

Any idea make wait=False option working in 'local mode'?

Thanks for reading.

0 Answers
Related