Using model for prediction in Vertex AI (Google Cloud Platform)

Question

I am following a tutorial of Vertex AI on google cloud, based on colab (text classification):

https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/official/automl/automl-text-classification.ipynb

and in particular I would like to understand how to use on GCP (Google Cloud Platform) a model trained in vertex-ai for thousands of predictions.

I think the interesting part of the tutorial about this topic is section "Get batch predictions from your model". However there they show a method that involves producing a bunch of files, one for each single text, and save all of them in a bucket on google cloud storage. These are the lines on the notebook where this is done:

instances = [
    "We hiked through the woods and up the hill to the ice caves",
    "My kitten is so cute",
]
input_file_name = "batch-prediction-input.jsonl"

...

# Instantiate the Storage client and create the new bucket
storage = storage.Client()
bucket = storage.bucket(BUCKET_NAME)

# Iterate over the prediction instances, creating a new TXT file
# for each.
input_file_data = []
for count, instance in enumerate(instances):
    instance_name = f"input_{count}.txt"
    instance_file_uri = f"{BUCKET_URI}/{instance_name}"

    # Add the data to store in the JSONL input file.
    tmp_data = {"content": instance_file_uri, "mimeType": "text/plain"}
    input_file_data.append(tmp_data)

    # Create the new instance file
    blob = bucket.blob(instance_name)
    blob.upload_from_string(instance)

input_str = "
".join([str(d) for d in input_file_data])
file_blob = bucket.blob(f"{input_file_name}")
file_blob.upload_from_string(input_str)

...

and after this they load the model and create a job:

job_display_name = "e2e-text-classification-batch-prediction-job"
model = aiplatform.Model(model_name=model_name)

batch_prediction_job = model.batch_predict(
    job_display_name=job_display_name,
    gcs_source=f"{BUCKET_URI}/{input_file_name}",
    gcs_destination_prefix=f"{BUCKET_URI}/output",
    sync=True,
)

My question is this. Is it really necessary to produce one single file for each sentence? If we have thousand of texts, this involves producing and saving on GCP thousand of small files. Doesn't this harm performance? Moreover, is the model still able to process "in batches" the input, like in a usual Tensorflow Model, taking advantage of vectorization?

user_14733669 · Answer

I checked the tutorial provided and the data used to train the model is per sentence. With regards to your questions:

Is it really necessary to produce one single file for each sentence?
- It really depends on what are you predicting if you are predicting sentences, paragraphs, etc. You can try passing a file with multiple sentences or paragraphs to test if the trained model can handle it.
- If you are not satisfied with the results, you can add more training data in multiple sentences or paragraphs (if that is your requirement) and retrain the model then test again until you are satisfied with the results.
If we have thousand of texts, this involves producing and saving on GCP thousand of small files. Doesn't this harm performance?
- Based from the batch prediction documentation, it is possible to enable scaling when generating batch prediction jobs. So this should handle the concern about the performance.
- If you use an autoscaling configuration, Vertex AI automatically scales your DeployedModel or BatchPredictionJob to use more prediction nodes when the CPU usage of your existing nodes gets high. Vertex AI scales your nodes based on CPU usage even if you have configured your prediction nodes to use GPUs; therefore if your prediction throughput is causing high GPU usage, but not high CPU usage, your nodes might not scale as you expect.
- Here is a sample code on how to define enable scaling on batch prediction jobs using the code from the tutorial:

job_display_name = "e2e-text-classification-batch-prediction-job-scale"
model = aiplatform.Model(model_name=model_name)

batch_prediction_job = model.batch_predict(
    job_display_name=job_display_name,
    gcs_source=f"{BUCKET_URI}/{input_file_name}",
    gcs_destination_prefix=f"{BUCKET_URI}/output",
    sync=True,
    machine_type='n1-highcpu-16', #define machine type
    starting_replica_count=1, 
    max_replica_count=10, # set max_replica_count > starting_replica_count to enable scaling
)

batch_prediction_job_name = batch_prediction_job.resource_name

I checked the generated logs and it took effect:

Moreover, is the model still able to process "in batches" the input, like in a usual Tensorflow Model, taking advantage of vectorization?
- I'm not familiar with Tensorflow model vectorization. It might be better to create a separate question for this so the community can contribute.

Using model for prediction in Vertex AI (Google Cloud Platform)

1 Answers