Real time barcode processing with Android Camera2 API and Firebase ML-Kit Vision

Viewed 1602

In my app rendering a preview while trying to process barcodes using the ML-Kit Vision library is really choppy. I think this is due to my lack of understanding of background theads and runnables.

I've setup my app following a few of Googles / Firebase quickstart guides and the Camera2 preview works fine.

I've also setup a onImageAvailableListener callback

/**
 * This a callback object for the [ImageReader]. "onImageAvailable" will be called when a
 * preview frame is ready to be processed.
 */
private val onImageAvailableListener = ImageReader.OnImageAvailableListener {
    // Processing code here
}

Here's my first attempt. Post to the background handler with the mediaImage and call image.close() in the BarcodeProcessor.

/**
 * This a callback object for the [ImageReader]. "onImageAvailable" will be called when a
 * preview frame is ready to be processed.
 */
private val onImageAvailableListener = ImageReader.OnImageAvailableListener {

    val mediaImage = it.acquireNextImage()
    backgroundHandler?.post(BarcodeProcessor(mediaImage, barcodeDetector))
}

internal class BarcodeProcessor(
    private val image: Image,
    private val barcodeDetector: FirebaseVisionBarcodeDetector
) : Runnable {

    override fun run() {
        Log.d(TAG, "run")
        try {
            val firebaseVisionImage = FirebaseVisionImage.fromMediaImage(image, 1)
            // ... more processing
        } catch (e: IOException) {
            Log.e(TAG, e.toString())
        } finally {
            image.close()
        }
    }

    companion object {
        /**
         * Tag for the [Log].
         */
        private val TAG = "BarcodeProcessor"
    }
}

Problem with this is the onImageAvailableListener posts in multiple images to the backgroundHandler without closing them in time. The val firebaseVisionImage = FirebaseVisionImage.fromMediaImage(image, 1) is really slow which is part of the issue.

Second Attempt

Here I try and close the image immediately after the backgroundHandler has been called. The issue with this approach is the image is already closed inside of the backgroundHandler

/**
 * This a callback object for the [ImageReader]. "onImageAvailable" will be called when a
 * preview frame is ready to be processed.
 */
private val onImageAvailableListener = ImageReader.OnImageAvailableListener {

    val mediaImage = it.acquireNextImage()
    backgroundHandler?.post(BarcodeProcessor(mediaImage, barcodeDetector))
    mediaImage.close()
}

internal class BarcodeProcessor(
    private val image: Image,
    private val barcodeDetector: FirebaseVisionBarcodeDetector
) : Runnable {

    override fun run() {
        Log.d(TAG, "run")
        try {
            val firebaseVisionImage = FirebaseVisionImage.fromMediaImage(image, 1)
            // ... more processing
        } catch (e: IOException) {
            Log.e(TAG, e.toString())
        } finally {

        }
    }

    companion object {
        /**
         * Tag for the [Log].
         */
        private val TAG = "BarcodeProcessor"
    }
}

Any help / tips / direction would be greatly appreciated.

1 Answers
Related