Abstract
I'm trying to test SageMaker Debugger examples in 'local mode' from amazon-sagemaker-examples. I'm trying to see the same debugging result in 'local mode' with AWS SageMaker Notebook instance.
What i've done
I put a few lines in one of the examples to use it in 'local mode' referenced from amazon-sagemaker-local-mode-example. The example is tf-mnist-builtin-rule.ipynb. Commented line is original code in the example
import subprocess
# import boto3
from sagemaker.local import LocalSession
from sagemaker.tensorflow import TensorFlow
instance_type = 'local'
try:
if subprocess.call("nvidia-smi") == 0:
instance_type = "local_gpu"
except:
pass
session = LocalSession()
session.config = {'local': {'local_code': True}}
# session = boto3.session.Session()
# region = session.region_name
role = 'arn:aws:iam::111111111111:role/service-role/AmazonSageMaker-ExecutionRole-20200101T000001'
estimator = TensorFlow(
# role=sagemaker.get_execution_role(),
role=role,
instance_count=1,
# instance_type="ml.p3.8xlarge",
instance_type=instance_type,
# image_uri=f"763104351884.dkr.ecr.{region}.amazonaws.com/tensorflow-training:2.3.1-gpu-py37-cu110-ubuntu18.04",
framework_version='2.4.1',
py_version="py37",
max_run=3600,
source_dir="./src",
entry_point="tf-resnet50-cifar10.py",
# Debugger Parameters
rules=built_in_rules,
sagemaker_session=session,
)
Problem:
I can't see debugging report in realtime unlike AWS SageMaker Notebook instance.
The example said wait=false option (see the figure) makes notebook proceed though it is still running but not working in 'local mode'.
Question:
Any idea make wait=False option working in 'local mode'?
Thanks for reading.
