Whenever I submit a training job to the AI platform, I have to wait around 5-10 minutes for my training job to start after it is queued. This happens when I submit a package for training as well as when I submit a docker image.
The logs go something like:
T0: "Job creation request has been successfully validated."
T0 + 1s: "Waiting for job to be provisioned."
T0 + 1s: "Job trainer_20200612_121714 is queued."
T0 + 3s: "Waiting for training program to start."
T0 + 7min: My training job finally starts
Is this completely on Google's side? Are there any settings on my end that can affect this?