Using onPreviewFrame To run ML models

Question

So i'm using the legacy Camera API (as far as I can tell) to get previewFrame call backs to then run a few machine learning models I have. I have confirmed that the machine learning models work when given a bitmap decoded when I take a picture via the onPictureTaken callback. Right now in the samples below, I am just simply testing on ML Kit's barcode scanner as a base case, but my custom models seemed to work fine with the onPictureTaken callback as well.

From what i've gathered, using onPreviewFrame isn't necessarily the best way to do this, but for the sake of having a quick sample play-around (and learning experience) I decided to just go this route. Based on everything i've tried from others having solutions online, I can't seem to get anything to work properly. The below code returns null:

@Override
public void onPreviewFrame(byte[] data, Camera camera) {
//        Log.d("onPreviewFrame bytes.length", String.valueOf(bytes.length));
//        final Bitmap bmp = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
//        Log.d("onPreviewFrame bmp.getHeight()", String.valueOf(bmp.getHeight()));

    Camera.Parameters parameters = camera.getParameters();
    int width = parameters.getPreviewSize().width;
    int height = parameters.getPreviewSize().height;


    Log.d("onPreviewFrame - width", String.valueOf(width));
    Log.d("onPreviewFrame - height", String.valueOf(height));
    Log.d("onPreviewFrame - parameters.getPreviewFormat()", String.valueOf(parameters.getPreviewFormat()));

    YuvImage yuv = new YuvImage(data, parameters.getPreviewFormat(), width, height, null);

    ByteArrayOutputStream out = new ByteArrayOutputStream();
    yuv.compressToJpeg(new Rect(0, 0, width, height), 100, out);

//
//        byte[] bytes = out.toByteArray();
//        final Bitmap bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);


    byte[] bytes = yuv.getYuvData();
    final Bitmap bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);

    extractBarcode(FirebaseVisionImage.fromBitmap(bitmap), bitmap);
}

Here's something else I tried:

@Override
public void onPreviewFrame(byte[] data, Camera camera) {
//        Log.d("onPreviewFrame bytes.length", String.valueOf(bytes.length));
//        final Bitmap bmp = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
//        Log.d("onPreviewFrame bmp.getHeight()", String.valueOf(bmp.getHeight()));

    Camera.Parameters parameters = camera.getParameters();
    int width = parameters.getPreviewSize().width;
    int height = parameters.getPreviewSize().height;


    Log.d("onPreviewFrame - width", String.valueOf(width));
    Log.d("onPreviewFrame - height", String.valueOf(height));

    YuvImage yuv = new YuvImage(data, parameters.getPreviewFormat(), width, height, null);

    ByteArrayOutputStream out = new ByteArrayOutputStream();
    yuv.compressToJpeg(new Rect(0, 0, width, height), 100, out);


    byte[] bytes = out.toByteArray();
    final Bitmap bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.length);

    extractBarcode(FirebaseVisionImage.fromBitmap(bitmap), bitmap);
}

Unfortunately I got this error:

ML Kit has detected that you seem to pass camera frames to the detector as a Bitmap object. This is inefficient. Please use YUV_420_888 format for camera2 API or NV21 format for (legacy) camera API and directly pass down the byte array to ML Kit.

with parameters.getPreviewFormat() returning 17 which is NV21. I also tried simply by changing that to ImageFormat.YUV_420_888 but that resulted in the below illegal argument exception:

only support ImageFormat.NV21 and ImageFormat.YUY2 for now

user_13546426 · Answer

Instead of using the Camera API, try using CameraX. It's easy to use and you can execute your code whenever a frame is received from the camera. While trying to integrate an ML model with the camera, I faced a similar error and then turned to CameraX.

Basically, we'll create an ImageAnalysis.Analyser class through which we would get the Image object ( frames ). Using an extension function, we will convert this Image object to a YuvImage.

You can follow this codelab to use CameraX to analyze frames. You will create a class that extends ImageAnalysis.Analyser class.

class FrameAnalyser() : ImageAnalysis.Analyzer {

    override fun analyze(image: ImageProxy?, rotationDegrees: Int) {
        val yuvImage = image?.image?.toYuv() // The extension function
    }
}

Create the extension function which transforms the Image to a YuvImage.

private fun Image.toYuv(): YuvImage {
    val yBuffer = planes[0].buffer
    val uBuffer = planes[1].buffer
    val vBuffer = planes[2].buffer
    val ySize = yBuffer.remaining()
    val uSize = uBuffer.remaining()
    val vSize = vBuffer.remaining()
    val nv21 = ByteArray(ySize + uSize + vSize)
    yBuffer.get(nv21, 0, ySize)
    vBuffer.get(nv21, ySize, vSize)
    uBuffer.get(nv21, ySize + vSize, uSize)
    val yuvImage = YuvImage(nv21, ImageFormat.NV21, this.width, this.height, null)
    return yuvImage
}

You can change the YUV Image format as required. Refer to these docs.

user_12991842 · Answer

Insted of directly passing FirebaseVisionImage

extractBarcode(FirebaseVisionImage.fromBitmap(bitmap), bitmap);

you can do it like this

var bitmap = toARGBBitmap(ocrBitmap)
  extractBarcode(FirebaseVisionImage.fromBitmap(bitmap), bitmap);

 private fun toARGBBitmap(img: Bitmap): Bitmap {
        return img.copy(Bitmap.Config.ARGB_8888, true)
    }

You can try this:)

Using onPreviewFrame To run ML models

2 Answers