Vertex AI endpoints - dataset creation after every batch prediction

user_15064662 0
Viewed 56

Vertex AI endpoints provide Batch Prediction functionality that is very useful to perform predictions on a large amount of data.

However, every time I make a new batch prediction the endpoint creates a new Dataset inside BigQuery. This is very uncomfortable because if I make 100 batch predictions per day, I will obtain 100 new Datasets inside BigQuery.

Is there a way to make all predictions converge inside the same Dataset? I mean, each prediction is a new Table inside a specific Dataset of BigQuery.

mlops google-cloud-vertex-ai google-cloud-platform python
1 Answers

Yes there is a way to make Vertex predictions in the same Dataset you need, to specify the Dataset ID in the OUTPUT_URI in your Batch Prediction.

bq://bqprojectId.bqDatasetId

If you just specify the Project ID, Vertex AI creates a new output dataset for you. Specifying the dataset ID is an optional resource.

You can also specify the table ID but this is not always the best.

bq://bqprojectId.bqDatasetId.bqTableId

You can see in this document the destination prefix while doing a batch prediction.

user_169293580
Related