I'm running a vertex batch job on a custom model with 676 instances in my jsonl.
I get results for all 676 instances but the job shows "Due to one or more errors, this batch prediction job was canceled on Aug 22, 2022 at 09:04PM"
The error file prediction.errors_stats-00000-of-00001 has 0 bytes and there is a directory:
beam-temp-prediction.errors_stats-e289aa8c228c11eda06642010a800fdc
with 4 files with 0 bytes in them.
34e19e3d-e717-4c6f-860c-5c2a177f1e93.prediction.errors_stats
7c9374c8-65f7-4a97-8a95-6068ba9cd936.prediction.errors_stats
981a634b-d27c-4285-b871-648b25ca87b8.prediction.errors_stats
f14dbc29-42d7-480e-8b79-95e799e132bf.prediction.errors_stats
Any ideas on why the job is showing failure but the results seem ok?
Job params:
{"displayName": <MY_DISPLAY_NAME>,
"model": <MY_MODEL>,
"inputConfig": {"instancesFormat": "jsonl",
"gcsSource": {"uris": ["gs://<MY_BUCKET>/MY_INSTANCES.jsonl"]}},
"outputConfig": {"predictionsFormat": "jsonl",
"gcsDestination": {"outputUriPrefix": "gs://<MY_OUTPUTS>/outputs/2022-08-23"}},
"dedicatedResources": {"machineSpec": {"machineType": "n1-standard-8",
"acceleratorType": null, "acceleratorCount": null},
"startingReplicaCount": 2},
"manualBatchTuningParameters": {"batch_size": 1}}
Note: I've also tried leaving out the 'startingReplicaCount and manualBatchTuningParameters.
I'm using a curl command to initiate the batch job:
curl -X POST \
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
-H "Content-Type: application/json; charset=utf-8" \
-d @$INPUT_JSON \
"https://$LOCATION-aiplatform.googleapis.com/v1/projects/$PROJECT/locations/$LOCATION/batchPredictionJobs"