When running tensorflow with REMOTE distributed command: after specifying "scale-tier STANDARD_1". Batch failed to run.... Now, I can ONLY run with simple SINGLE NODE "scale-tier=BASIC"
gcloud ml-engine jobs submit training census_20171005_162623
--stream-logs --scale-tier STANDARD_1 --staging-bucket gs://dextest --runtime-version 1.2 --job-
dir gs://dextest/census_20171005_162623 --module-name trainer.task --package-path trainer/ --
region us-central1 -- --train-files
gs://cloudml-public/census/data/adult.data.csv --eval-files
gs://cloudml-public/census/data/adult.test.csv --train-steps 1000
--eval-steps 100
The error I am getting is
The replica worker 1 exited with a non-zero status of 1. Termination reason: Error.
From the log:
Retrying after gsutil exception Command '['gsutil', '-q', 'cp',
u'gs://dextest/census_20171005_161531/2211a814b974edbc3defee855046dd8e801393b7ff8154b084b081322167fe90/trainer-0.0.0.tar.gz', u'trainer-0.0.0.tar.gz']' returned non-zero exit status 1.
The Master did SUCCESSFULLY initialized and copued the package "trainer-0.0.0.tar.gz" However, issue happened when replicas copying package for the run. It seems that the ML workflow FAILED to handle the cleanup in replicas.... The replicas tried to clean the job directory "gs://dextest/census_20171005_162623" again before running.
the error happened after master copy the package and the replicas FAILED to pick up the package for running.
I CommandException: No URLs matched: gs://dextest/census_20171005_161531/2211a814b974edbc3defee855046dd8e801393b7ff8154b084b081322167fe90/trainer-0.0.0.tar.gz E Retrying after gsutil exception Command '['gsutil', '-q', 'cp', u'gs://dextest/census_20171005_161531/2211a814b974edbc3defee855046dd8e801393b7ff8154b084b081322167fe90/trainer-0.0.0.tar.gz', u'trainer-0.0.0.tar.gz']' returned non-zero exit status 1.
undefined