Best usage of CameraX for MLKit Text Recognition on Android

user_1478296 0
Viewed 1524

I need to implement text recognition using MLKit on Android and I have decided to use the new CameraX api as the camera lib. I am struggling with the correct "pipeline" of classes or data flow of the image because CameraX is quite new and not many resources is out there. The use case is that I take the picture, crop it in the middle by some bounds that are visible in the UI and then pass this cropped image to the MLKit that will process the image.

Given that, is there some place for ImageAnalysis.Analyzer api? From my understanding this analyzer is used only for previews and not the captured image.

My first idea was to use takePicture method that accepts OnImageCapturedCallback but when I've tried access eg. ImageProxy.height the app crashed with an exception java.lang.IllegalStateException: Image is already closed and I could not find any fix for that.

Then I've decided to use another overload of takePicture method and now I save image to the file, then read it to Bitmap, crop this image and now I have an image that can be passed to MLKit. But when I take a look at FirebaseVisionImage that is passed to FirebaseVisionTextRecognizer it has a factory method to which I can pass the Image that I get from OnImageCapturedCallback which seems that I am doing some unnecessary steps.

So my questions are:

  1. Is there some class (CaptureProcessor?) that will take care of the cropping of taken image? I suppose that then I could use OnImageCapturedCallback where I would receive already cropped image.
  2. Should I even use ImageAnalysis.Analyzer if I am not doing realtime processing and I am doing post processing?

I suppose that I can achieve what I want with my current approach but I am feeling that I could use much more of CameraX than I currently am.

Thanks!

text-recognition firebase-mlkit android-camerax android
1 Answers

Is there some class (CaptureProcessor?) that will take care of the cropping of taken image?

You can set the crop aspect ratio after you build the ImageCapture use case by using the setCropAspectRatio(Rational) method. This method crops from the center of the rotated output image. So basically what you'd get back after calling takePicture() is what I think you're asking for.

Should I even use ImageAnalysis.Analyzer if I am not doing realtime processing and I am doing post processing?

No, it wouldn't make sense in your scenario. As you mentioned, only when doing real-time image processing would you want to use ImageAnalysis.Analyzer.

ps: I'd be interested in seeing the code you use for takePicture() that caused the IllegalStateException.

[Edit]

Taking a look at your code

imageCapture?.takePicture(executor, object : ImageCapture.OnImageCapturedCallback() {
    override fun onCaptureSuccess(image: ImageProxy) {
        // 1
        super.onCaptureSuccess(image)

        // 2
        Log.d("MainActivity", "Image captured: ${image.width}x${image.height}")
    }
})

At (1), if you take a look at super.onCaptureSuccess(imageProxy)'s implementation, it actually closes the imageProxy passed to the method. Accessing the image's width and height in (2) throws an exception, which is normal -since the image has been closed-. The documentation states:

The application is responsible for calling ImageProxy.close() to close the image.

So when using this callback, you should probably not call super..., just use the imageProxy, then before returning from the method, manually close it (ImageProxy.close()).

user_47182830
Related