Android camera2 api. Setting multiple ImageReader surfaces gives blank output

I have a camera2 implementation. The current setup is, it uses a texture view surface to display the actual camera view and an ImageReader surface for capturing images.

Now I want to capture preview frames as well. So I tried adding a new ImageReader surface for capturing frames. But when I add that surface to createCaptureSession request, the screen goes blank. What could possibly be wrong? Below is the code that I use to add surfaces to createCaptureSession

val surface = preview.surface
            ?: throw CameraAccessException(CameraAccessException.CAMERA_ERROR)

val previewIRSurface = previewImageReader?.surface
            ?: throw CameraAccessException(CameraAccessException.CAMERA_ERROR)

val captureSurface = captureImageReader?.surface
            ?: throw CameraAccessException(CameraAccessException.CAMERA_ERROR)

try {
    val template = if (zsl) CameraDevice.TEMPLATE_ZERO_SHUTTER_LAG else CameraDevice.TEMPLATE_PREVIEW

    previewRequestBuilder = camera?.createCaptureRequest(template)
            ?.apply { addTarget(surface) }
            ?: throw CameraAccessException(CameraAccessException.CAMERA_ERROR)

    val surfaces: ArrayList<Surface> = arrayListOf(surface, previewIRSurface, captureSurface)

    camera?.createCaptureSession(surfaces, sessionCallback, backgroundHandler)

} catch (e: CameraAccessException) {
    throw RuntimeException("Failed to start camera session")
}

The initialization of ImageReaders is like this.

private fun prepareImageReaders() {

    val largestPreview = previewSizes.sizes(aspectRatio).last()

    previewImageReader?.close()

    previewImageReader = ImageReader.newInstance(
            largestPreview.width,
            largestPreview.height,
            internalOutputFormat,
            4 // maxImages
    ).apply { setOnImageAvailableListener(onPreviewImageAvailableListener, backgroundHandler) }

    val largestPicture = pictureSizes.sizes(aspectRatio).last()

    captureImageReader?.close()

    captureImageReader = ImageReader.newInstance(
            largestPicture.width,
            largestPicture.height,
            internalOutputFormat,
            2 // maxImages
    ).apply { setOnImageAvailableListener(onCaptureImageAvailableListener, backgroundHandler) }
}

More clarifications about the parameters used above:

• internalOutput format is either ImageFormat.JPEG or ImageFormat.YUV_420_888.
• Image sizes are based on best possible sizes
• It works good with either of the image readers individually but as soon as I add both together, blank screen!
• Testing on Samsung Galaxy S8 with Android Oreo (8.0)

The original code is here https://github.com/pvasa/cameraview-ex/blob/development/cameraViewEx/src/main/api21/com/priyankvasa/android/cameraviewex/Camera2.kt