cv::cvtColor without allocating memory issue

I am trying to convert a 320x240 image from NV21 into a RGBA one using cv::cvtColor, without allocating new memory.

So I tried various ways of achieving this. I know that cv::cvtColor calls cv::Mat.create(..) to allocate memory for the output Mat, and I've seen that cv::Mat.create(...) only allocates new memory if the size & type of the requested Mat differs from its current values.

Given, these, the following:

//(1)
cv::Mat frame(height + height / 2, width, CV_8UC1, pointerToNV21Data);
//(2)
cv::cvtColor(frame, frame, cv::COLOR_YUV2RGBA_NV21);

Allocates a new continuous memory region of 320 * 240 * 4 channels = 307200 bytes (previously free memory), and discards the previously allocated 320 * 360 * 1 channel = 115200 bytes.

When cv::cvtColor internally calls cv::Mat.create(...), the new memory block is allocated because:

• before cv::cvtColor, frame was a cv::Mat of size 320x360, with 1 channel
• cv::cvtColor requires it to be a cv::Mat of size 320x240 with 4 channels

So my most optimistic attempt to solving this was to do the following:

//allocate a byte buffer of 307200 bytes
uchar largeBuffer[307200];

//make sure the 115200 NV21 image bytes are written to the buffer
writeNV21DataToBuffer(largeBuffer);

//create cv::Mat that wraps "largeBuffer" as a NV21 image
cv::Mat frameNV21(height + height / 2, width, CV_8UC1, largeBuffer);

//create a second cv::Mat that wraps around the same "largeBuffer", 
//but this time consider it as pointing to a RGBA image
cv::Mat frameRGBA(height, width, CV_8UC4, largeBuffer);

//call cv::cvtColor
cv::cvtColor(frameNV21, frameRGBA, cv::COLOR_YUV2RGBA_NV21);

By doing this, indeed no new memory is allocated. Printing cv::Mat.dataStart & cv::Mat.dataStart of frameNV21, frameRGBA before cv::cvtColor and frameRGBA after cv::cvtColor confirms that the same memory region was used, and that no new memory was allocated for the result.

However...the conversion itself is not done properly, the result looking like this, when displayed on the screen:

At this point, I'm out of ideas. Is my last approach inherently wrong, or is there no way to achieve this?