Google Vertex AI image AutoML classification when an important image feature is text inside the image

user_3033040 0
Viewed 296

I'd like to do image classification. In my dataset, despite the fact that images features is a strong component for this classification (colors, shapes, etc), some categories of images will be hard to distinguish without interpreting the text inside the image.

I don't think VertexAI/AutoML will use pre-trained models in order to facilitate classification if in some case the only difference is the text. I know Google Vision/OCR is capable of doing such extraction. But is there a way to do image classification (VertexAI/AutoML) using Google Cloud Vision extraction as an additional image feature?

Currently my project uses 3 models (no google cloud):

  • model 1: classify an image using images features
  • model 2: classify an image, only using OCR + regex (same categories)
  • model 3: combine both models and decide when to use model 1 or model 2

I'd like to switch to Vertex AI the following will improve my project quality for the following:

  • AutoML classification seems very good for model 1
  • I need to use a tool to manage my datasets (Vertex AI managed dataset)
  • Vertex AI has interesting pipeline training features

If it is confirmed that AutoML won't perform well if some images categories only differs in the text, I would recreate a similar 3-tier models using Vertex AI custom training scripts. I can easily create model 1 with VertexAI/AutoML. However I have no idea if:

  • I can create model 2 with a vertex ai custom training script using google cloud vision/ocr to do image classification
  • I can create model 3 that would use models 1 and 2 created by vertex ai.

Could you give me recommendations on how to achieve that using Google Cloud Platform?

google-cloud-automl google-cloud-vision google-cloud-vertex-ai google-cloud-platform ocr
1 Answers

For this purpose, I recommend you the following:

1. model 2:

  • Keep your images in a GCS.
  • Use the Detect text in images  |  Cloud Vision API to generate your dataset (text) {"gcs":"gs://path_to_image/image_1","text":["text1"...]}.
  • Use AutoML on this text dataset processed by vision api or just use a regexp on this data or insert into a bigquery dataset and query on it, and so on...

1. model 3:

  • I would follow a similar approach, processing the images using the cloud vision API and generating a text dataset, but this time, the images that dont have any text on it, will generate a dataset with the "text" field empty {"gcs":"gs://path_to_image/image_2","text":[]}. Your own script can exclude the data with text and generate a dataset for the model 2, and a dataset for the model 1.

I see that your models 2 and 3 are not strictly classifications. Model 2 is a ocr problem, and them you process the output data. The model 3 is basically process your data and separate the proper datasets.

I hope this insight may help you.

user_132487380
Related