How to reproduce the conversion from YUV to RGB done by the iOS's Accelerate module?

Viewed 106

I am trying to convert a YUV frame to RGB that was captured by system broadcast in iOS device. Please note, I want to extract the YUV frame out of the device and convert it to RGB outside of the device, instead of converting it in the device.

First, I was able to extract the YUV frame as two grayscale images. Luma is extracted as an 8-bit and Chroma as a 16-bit half-size grayscale image.

Luma | Chroma

Second, I was also able to convert to RGB using the Accelerate module in the device (which is exactly matched to the image I need). This conversion was done from the two png files above, So I assume that I have successfully extracted everything I need.

Now, I am trying to reproduce this conversion outside of the device but am struggling.

The image on the left below was generated by the Accelerate module, and the image on the right was reproduced in python. There are noticeable differences if you compare the two pieces overlaid on top of each other. I don't care about tiny differences such as rounding errors, but I would like to get a result that is close enough that the differences are not visually noticeable.

Expected | Actual

Question

How can I reproduce the image on the left correctly?


Working conversion code

It's a bit complicated (sorry for that), I am using Objective-C++ to load the images and Swift to do the conversion.

Here is Objective-C++ part.

// You must first import the OpenCV.
#import <opencv2/opencv.hpp>
#import <opencv2/imgcodecs/ios.h>

// Then import others.
#import <Foundation/Foundation.h>
#import <UIKit/UIKit.h>

// 16bit-grayscale version of MatToCGImage.
CGImageRef CV16UC1MatToCGImage(const cv::Mat& image) {
    NSData *data = [NSData dataWithBytes:image.data
                                  length:image.step.p[0] * image.rows];
    CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceGray();
    CGDataProviderRef provider = CGDataProviderCreateWithCFData((__bridge CFDataRef)data);
    CGBitmapInfo bitmapInfo = kCGImageAlphaNone | kCGImageByteOrderDefault;
    CGImageRef imageRef = CGImageCreate(image.cols,
                                        image.rows,
                                        8 * image.elemSize1(),  // bitsPerComponent
                                        8 * image.elemSize(),  // bitsPerPixel
                                        image.step.p[0],  // bytesPerRow
                                        colorSpace,
                                        bitmapInfo,
                                        provider,
                                        NULL,
                                        false,
                                        kCGRenderingIntentDefault
                                        );
    CGDataProviderRelease(provider);
    CGColorSpaceRelease(colorSpace);
    return imageRef;
}

// Use like this
cv::Mat chromaMat = cv::imread(path_to_chroma, -1);
CGImageRef chromaCGImage = CV16UC1MatToCGImage(chromaMat);

cv::Mat lumaMat = cv::imread(path_to_luma, -1);
CGImageRef lumaCGImage = MatToCGImage(lumaMat);

Here is Swift part that convert Luma/Chroma to RGB.

    public func convert(lumaImage: CGImage, chromaImage: CGImage) -> CGImage? {
        var pixelRange = vImage_YpCbCrPixelRange(
            Yp_bias: 0,
            CbCr_bias: 128,
            YpRangeMax: 255,
            CbCrRangeMax: 255,
            YpMax: 255,
            YpMin: 0,
            CbCrMax: 255,
            CbCrMin: 0
        )

        var conversionInfo = vImage_YpCbCrToARGB()

        guard vImageConvert_YpCbCrToARGB_GenerateConversion(
            kvImage_YpCbCrToARGBMatrix_ITU_R_709_2,
            // Tried 601 too, but it's not the image I need and can't reproduce anyway.
            // kvImage_YpCbCrToARGBMatrix_ITU_R_601_4,
            &pixelRange,
            &conversionInfo,
            kvImage420Yp8_CbCr8, // from
            kvImageARGB8888, // to
            vImage_Flags(kvImageNoFlags)
        ) == kvImageNoError else {
            return nil
        }

        guard var lumaBuffer = try? vImage_Buffer(cgImage: lumaImage) else {
            return nil
        }

        guard var chromaBuffer = try? vImage_Buffer(cgImage: chromaImage) else {
            return nil
        }

        var argbBuffer = vImage_Buffer()
        guard vImageBuffer_Init(
            &argbBuffer,
            lumaBuffer.height,
            lumaBuffer.width,
            32,
            vImage_Flags(kvImageNoFlags)
        ) == kvImageNoError else {
            return nil
        }

        guard vImageConvert_420Yp8_CbCr8ToARGB8888(
            &lumaBuffer, // in
            &chromaBuffer, // in
            &argbBuffer, // out
            &conversionInfo,
            nil,
            255,
            vImage_Flags(kvImageNoFlags)
        ) == kvImageNoError else {
            return nil
        }

        guard let context = CGContext(
            data: argbBuffer.data,
            width: Int(argbBuffer.width),
            height: Int(argbBuffer.height),
            bitsPerComponent: 8,
            bytesPerRow: argbBuffer.rowBytes,
            space: CGColorSpaceCreateDeviceRGB(),
            bitmapInfo: CGImageAlphaInfo.premultipliedFirst.rawValue | CGBitmapInfo.byteOrder32Big.rawValue
        ) else {
            return nil
        }

        let rgbImage = context.makeImage()
        
        return rgbImage
    }

Failed attempt

The code below generates the image on the right, which is incorrect.

def split_16UC1_to_8UC2(source):
    assert source.dtype == np.uint16 and source.ndim == 2
    low = source.astype(np.uint8)
    high = (source >> 8).astype(np.uint8)
    return high, low


def convert():
    luma = cv2.imread("./luma.png", -1)
    chroma = cv2.imread("./chroma.png", -1)
    v, u = split_16UC1_to_8UC2(chroma)
    h, w = luma.shape
    yuv = np.zeros([int(h * 1.5), w], dtype=np.uint8)
    yuv[:h, ...] = luma
    yuv[h:, ..., ::2] = u
    yuv[h:, ..., 1::2] = v
    rgb = cv2.cvtColor(yuv, cv2.COLOR_YUV2BGR_NV12)
    cv2.imwrite("./_rgb.png", rgb)

I also tried this nice library, but it did not make much difference.

0 Answers
Related