Incorrect image converting YUV_420_888 into Bitmaps under Android camera2

Question

I’m trying to convert YUV_420_888 images into bitmaps, coming from the camera2 preview. But the output image has incorrect colors.

Next is the test code I’m running to generate the bitmap. Is test code only, so please don’t do any code review about not relevant factors such as the bitmap is being recycled, or the RenderScript is continuously been created. This code is just to test the conversion from YUV to RGB and nothing more.

Other factors, the code is meant to run from API 22 and above, therefore using RenderScript specific ScriptIntrinsicYuvToRGB should be sufficient, without having to use old manual conversions which where necessary only in previous Android versions due to lack of proper YUV_420_888 support.

As RenderScript already offers a dedicated ScriptIntrinsicYuvToRGB which is meant to handle all types of YUV conversions, I think the problem could be in how I get the YUV byte data from the Image object, but I can't figure where the issue is.

To view the output bitmap in Android Studio, place a breakpoint in bitmap.recycle(), so before it gets recycled you can look at it in the Variables Debug Window by using the “view bitmap” option.

Please let me know if anyone can spot what’s wrong with the conversion:

@Override
public void onImageAvailable(ImageReader reader)
{
    RenderScript rs = RenderScript.create(this.mContext);

    final Image image = reader.acquireLatestImage();

    final Image.Plane[] planes = image.getPlanes();
    final ByteBuffer planeY = planes[0].getBuffer();
    final ByteBuffer planeU = planes[1].getBuffer();
    final ByteBuffer planeV = planes[2].getBuffer();

    // Get the YUV planes data

    final int Yb = planeY.rewind().remaining();
    final int Ub = planeU.rewind().remaining();
    final int Vb = planeV.rewind().remaining();

    final ByteBuffer yuvData = ByteBuffer.allocateDirect(Yb + Ub + Vb);

    planeY.get(yuvData.array(), 0, Yb);
    planeU.get(yuvData.array(), Yb, Vb);
    planeV.get(yuvData.array(), Yb + Vb, Ub);

    // Initialize Renderscript

    Type.Builder yuvType = new Type.Builder(rs, Element.YUV(rs))
            .setX(image.getWidth())
            .setY(image.getHeight())
            .setYuvFormat(ImageFormat.YUV_420_888);

    final Type.Builder rgbaType = new Type.Builder(rs, Element.RGBA_8888(rs))
            .setX(image.getWidth())
            .setY(image.getHeight());

    Allocation yuvAllocation = Allocation.createTyped(rs, yuvType.create(), Allocation.USAGE_SCRIPT);
    Allocation rgbAllocation = Allocation.createTyped(rs, rgbaType.create(), Allocation.USAGE_SCRIPT);

    // Convert

    yuvAllocation.copyFromUnchecked(yuvData.array());

    ScriptIntrinsicYuvToRGB scriptYuvToRgb = ScriptIntrinsicYuvToRGB.create(rs, Element.YUV(rs));
    scriptYuvToRgb.setInput(yuvAllocation);
    scriptYuvToRgb.forEach(rgbAllocation);

    // Get the bitmap

    Bitmap bitmap = Bitmap.createBitmap(image.getWidth(), image.getHeight(), Bitmap.Config.ARGB_8888);
    rgbAllocation.copyTo(bitmap);

    // Release

    bitmap.recycle();

    yuvAllocation.destroy();
    rgbAllocation.destroy();
    rs.destroy();

    image.close();
}

user_6456129 · Answer

RenderScript support YUV_420_888 as ScriptIntrinsicYuvToRGB's source

create Allocation & ScriptIntrinsicYuvToRGB

RenderScript renderScript = RenderScript.create(this);
ScriptIntrinsicYuvToRGB mScriptIntrinsicYuvToRGB = ScriptIntrinsicYuvToRGB.create(renderScript, Element.YUV(renderScript));
Allocation mAllocationInYUV = Allocation.createTyped(renderScript, new Type.Builder(renderScript, Element.YUV(renderScript)).setYuvFormat(ImageFormat.YUV_420_888).setX(480).setY(640).create(), Allocation.USAGE_IO_INPUT | Allocation.USAGE_SCRIPT);
Allocation mAllocationOutRGB = Allocation.createTyped(renderScript, Type.createXY(renderScript, Element.RGBA_8888(renderScript), 480, 640), Allocation.USAGE_SCRIPT | Allocation.USAGE_IO_OUTPUT);

set Allocation.getSurface() to receive image data from camera

final CaptureRequest.Builder captureRequest = session.getDevice().createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
captureRequest.addTarget(mAllocationInYUV.getSurface());

output to a TextureView or ImageReader or SurfaceView

mAllocationOutRGB.setSurface(new Surface(mTextureView.getSurfaceTexture()));
mAllocationInYUV.setOnBufferAvailableListener(new Allocation.OnBufferAvailableListener() {
    @Override
    public void onBufferAvailable(Allocation a) {
        a.ioReceive();
        mScriptIntrinsicYuvToRGB.setInput(a);
        mScriptIntrinsicYuvToRGB.forEach(mAllocationOutRGB);
        mAllocationOutRGB.ioSend();
    }
});

user_192373 · Answer

There is no direct way to copy a YUV_420_888 camera frame in into an RS Allocation. Actually, as of today, Renderscript does not support this format.

If you know that, under the hood, your frame is NV21 or YV12 - you can copy the entire ByteBuffer to an array and pass it to RS Allocation.

Incorrect image converting YUV_420_888 into Bitmaps under Android camera2

3 Answers