How to know when batch cloud prediction ends

Question

I am doing an application that receives some data, treats it then creates a prediction-ai batch job. After the predictions are done, I am supposed to take them all and merge them with a previous file. The batch prediction writes to a bucket, let's call is gs://predictions.

At the moment, I have a cloud function that gets triggered whenever gs://predictions is written to. However, the batch prediction job streams data into the file, updating such file many times when there are lots of predictions to be made. This means that my cloud function gets triggered many times when I just wanted it to be called when the job was finished.

To overcome this, right now the cloud function gets called and then checks whether the job is finished or not. If it is, process the file; if not, let it slide. This, of course, brings a lot of unnecessary processing (and unnecessary code!) :-(

What would really help me here: can the batch job somehow write to Pub/Sub when it is done? Or even better, can it use a webhook so that when it is finished itself calls my cloud function? I tried to look into the docs, but could not find anything.

Is there any other proposed solution for this?

user_11372593 · Answer

You can create a log sink into PubSub and filter the logs on this custom filter:

resource.type="ml_job"
textPayload="Job completed successfully."

Then, when a batch job completed, a log trace is printed and a message is posted into the PubSub topic.

user_4407260 · Answer

Refer to docs Google Cloud Training job docs In order get AI Platform job status ,you can use REST API's provided by GCP ,refer to jobs AI Platform jobs docs

GET Request body:

{
  "jobId": string,
  "createTime": string,
  "startTime": string,
  "endTime": string,
  "state": enum (State),  # This Field is what will tell you status of job
  "errorMessage": string,
  "labels": {
    string: string,
    ...
  },
  "etag": string,

  // Union field input can be only one of the following:
  "trainingInput": {
    object (TrainingInput)
  },
  "predictionInput": {
    object (PredictionInput)
  }
  // End of list of possible types for union field input.

  // Union field output can be only one of the following:
  "trainingOutput": {
    object (TrainingOutput)
  },
  "predictionOutput": {
    object (PredictionOutput)
  }
  // End of list of possible types for union field output.
}

State Field is enum

Enums
STATE_UNSPECIFIED   The job state is unspecified.
QUEUED              The job has been just created and processing has not yet begun.
PREPARING           The service is preparing to run the job.
RUNNING             The job is in progress.
SUCCEEDED           The job completed successfully.
FAILED              The job failed. errorMessage should contain the details of the failure.
CANCELLING          The job is being cancelled. errorMessage should describe the reason for the cancellation.
CANCELLED           The job has been cancelled. errorMessage should describe the reason for the cancellation.

Note: All this can be done via python or any other language of choice. Using Python GCP Client Libraries

How to know when batch cloud prediction ends

2 Answers