How to convert ByteBuffer to bitmap and vice versa without dropping the frames?

Viewed 351

I have a ByteBuffer:

var rgbaData = JniUtil.nativeAllocateByteBuffer(frame.width * frame.height * 4)

where the frame is the VideoFrame obtained from live streaming media. I want to apply the HighPassSkinSmoothing filter to the bitmap obtained from the above's buffer, convert the bitmap back to ByteArray, put it in the ByteBuffer and then release it back to the live streaming media. I have done the following to convert the buffer to bitmap, apply the filter and then again convert the bitmap to the ByteArray, and put it in the buffer.

For applying the HighPassSkinSmoothing Filter:

val bitmap = getBitmapFromBuffer(frame.width, frame.height, rgbaData)

amniXSkinSmooth.storeBitmap(bitmap, false)
amniXSkinSmooth.initSdk()
amniXSkinSmooth.startSkinSmoothness(600F)
amniXSkinSmooth.startSkinWhiteness(2F)
//sometimes "amniXSkinSmooth.bitmapAndFree" is also null, might be due to wrong conversion process
rgbaData.put(getByteArrayFromBitmap(amniXSkinSmooth.bitmapAndFree))
amniXSkinSmooth.unInitSdk();

Converting ByteBuffer to Bitmap:

private fun getBitmapFromBuffer(width: Int, height: Int, buffer: ByteBuffer): Bitmap {
    buffer.rewind()
    val bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888)
    bitmap.copyPixelsFromBuffer(buffer)
    return bitmap
}

Convert Bitmap back to ByteArray:

private fun getByteArrayFromBitmap(b: Bitmap): ByteArray {
  val size: Int = b.byteCount
  val buffer = ByteBuffer.allocate(size)
  val bytes = ByteArray(size)
  b.copyPixelsToBuffer(buffer)
  buffer.rewind()
  buffer.get(bytes)
  return bytes
}

By doing this, I am losing the frames and the live streaming is also lagging after applying the filter. Is there any workaround to this so that the frames wouldn't drop and the live streaming also wouldn't lag?

If you guys have any idea or knowledge regarding the conversion process or anything, your help would be much much appreciated. Thank you!!

1 Answers

I think we can use a memory mapped file e.g. MappedByteBuffer that can serve as a store to hold the rgb byte buffer. Now we can use a windowing mechanism e.g. sliding window/ tumbling window and process the data as per the windowing method. This way we can process the streaming data. Once the processing is done we can delete the underlying file for the mappedbytebuffer. Also the window can hold the start/end address of the mappedbytebuffer. Moreover the exact size of the window may depend on the bandwidth of the network.

Related