Currently, I'm having no luck getting a local compute target to run in WSL. I can happily execute my training script, and I can execute my driver, but the job never leaves the starting state. I've enabled 'node' in the Firewall - not sure if that is related - it popped up at the same time.
Driver Code:
ws = Workspace.from_config()
experiment = Experiment(workspace=ws, name='mnist-training-local')
env = Environment("system-managed-env")
env.python.user_managed_dependencies = True
config = ScriptRunConfig(source_directory='src', script='hello.py', environment=env)
run = experiment.submit(config)
run.wait_for_completion(show_output=True)
There is a diagnostic link output -
"data_container_id": "dcid.mnist-training-local_1605046048_2a328e04",
"diagnostics_uri": "https://eastus2.experiments.azureml.net/execution/v1.0/subscriptions/c14a37bd-a658-463c-9d44-9a9326fe5fbe/resourceGroups/TutorialResourceGroup/providers/Microsoft.MachineLearningServices/workspaces/TutorialWorkspace/experiments/mnist-training-local/runId/mnist-training-local_1605046048_2a328e04/diagnostics"
But the link is never active.
Viewing the job through the portal results in the job just staying in the 'Starting' state. It's like there's a firewall issue between WSL and Azure, but I'm not sure where.
I have tried stopping the Windows Defender entirely though - no luck. I suspect that there's an incoming connection that is not getting routed to WSL correctly?
'top' shows no real activity on the system, the system just is ... waiting. For eternity.
This is the first step of the following Azure ML notebook: https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/training/train-on-local/train-on-local.ipynb
User managed pre-reqs.
I can in fact, submit from a DOS (Powershell) terminal:
Web View: https://ml.azure.com/experiments/mnist-training-local/runs/mnist-training-local_1605107830_4f84d648?wsid=/subscriptions/c14a37bd-a658-463c-9d44-9a9326fe5fbe/resourcegroups/TutorialResourceGroup/workspaces/TutorialWorkspace
Streaming azureml-logs/60_control_log.txt
=========================================
[2020-11-11T15:17:14.044950] Using urllib.request Python 3.0 or later
Streaming log file azureml-logs/60_control_log.txt
Running: ['cmd.exe', '/c', 'C:\\Users\\allan\\AppData\\Local\\Temp\\azureml_runs\\mnist-training-local_1605107830_4f84d648\\azureml-environment-setup/conda_env_checker.bat']
<snip>
After variable expansion, calling script [ hello.py ] with arguments: []
Script type = None
Hello world!
Starting the daemon thread to refresh tokens in background for process with pid = 21880
Thanks in advance.