capture overlay using PreviewView of cameraX

Viewed 5005

I'm trying to capture a picture with overlay included in image capture. I was able to set overlay to previewView using cameraView.overlay.add(binding.textView). How ever, it did not save when trying to save an image with imageCapture Only the picture was saved not the overlay. How do I save an image with overlay included using PreviewView of camera x.

Please don't mark this as duplicate. I researched a lot and most of the example online are using the old camera api which does not apply to camera x library. Any help is appreciated. Thanks in advance.

Here is my code


           <FrameLayout
                android:id="@+id/camera_wrapper"
                android:layout_width="match_parent"
                android:layout_height="0dp"
                app:layout_constraintTop_toTopOf="@id/space1"
                app:layout_constraintBottom_toBottomOf="@id/space">
    
                <androidx.camera.view.PreviewView
                    android:id="@+id/camera_view"
                    android:layout_width="match_parent"
                    android:layout_height="match_parent" />
    
                <TextView
                    android:id="@+id/text_view"
                    android:layout_width="wrap_content"
                    android:layout_height="wrap_content"
                    android:layout_gravity="center"
                    android:text="Hello world"
                    android:textSize="42sp"
                    android:textColor="@android:color/holo_green_dark"/>
    
            </FrameLayout>

  private lateinit var outputDirectory: File
  private lateinit var cameraExecutor: ExecutorService
  private var preview: Preview? = null
  private var lensFacing: Int = CameraSelector.LENS_FACING_FRONT
  private var imageCapture: ImageCapture? = null
  private var camera: Camera? = null
  private var cameraProvider: ProcessCameraProvider? = null

override fun onViewCreated(view: View, savedInstanceState: Bundle?) {
    super.onViewCreated(view, savedInstanceState)
    outputDirectory = getOutputDirectory()
    cameraExecutor = Executors.newSingleThreadExecutor()
  }

  private fun setupCamera() {
    val cameraProviderFuture = ProcessCameraProvider.getInstance(requireContext())

    cameraProviderFuture.addListener(
      Runnable {
        // Used to bind the lifecycle of cameras to the lifecycle owner
        cameraProvider = cameraProviderFuture.get()

        // Get screen metrics used to setup camera for full screen resolution
        val metrics = DisplayMetrics().also { binding.cameraView.display.getRealMetrics(it) }
        Timber.d("Screen metrics: ${metrics.widthPixels} x ${metrics.heightPixels}")

        val screenAspectRatio = aspectRatio(metrics.widthPixels, metrics.heightPixels)
        Timber.d("Preview aspect ratio: $screenAspectRatio")

        val rotation = binding.cameraView.display.rotation

        // CameraProvider
        val cameraProvider = cameraProvider
          ?: throw IllegalStateException("Camera initialization failed.")

        // CameraSelector
        val cameraSelector = CameraSelector.Builder().requireLensFacing(lensFacing).build()

        // add text overlay *---------*
        binding.cameraView.overlay.add(binding.textView)

        // Preview
        preview = Preview.Builder()
          // We request aspect ratio but no resolution
          .setTargetAspectRatio(screenAspectRatio)
          // Set initial target rotation
          .setTargetRotation(rotation)
          .build()

        // ImageCapture
        imageCapture = ImageCapture.Builder()
          .setCaptureMode(ImageCapture.CAPTURE_MODE_MINIMIZE_LATENCY)
          // We request aspect ratio but no resolution to match preview config, but letting
          // CameraX optimize for whatever specific resolution best fits our use cases
          .setTargetAspectRatio(screenAspectRatio)
          // Set initial target rotation, we will have to call this again if rotation changes
          // during the lifecycle of this use case
          .setTargetRotation(rotation)
          .build()

        // Must unbind the use-cases before rebinding them
        cameraProvider.unbindAll()

        try {
          // A variable number of use-cases can be passed here -
          // camera provides access to CameraControl & CameraInfo
          camera = cameraProvider.bindToLifecycle(this, cameraSelector, preview, imageCapture)
          // Attach the viewfinder's surface provider to preview use case
          preview?.setSurfaceProvider(binding.cameraView.surfaceProvider)
        } catch (exc: Exception) {
          Toast.makeText(requireContext(), "Something went wrong. Please try again.", Toast.LENGTH_SHORT).show()
          findNavController().navigateUp()
        }
      },
      ContextCompat.getMainExecutor(requireContext())
    )
  }

private fun takePhoto() {
    imageCapture?.let { imageCapture ->

      // Create output file to hold the image
      val photoFile = createFile(outputDirectory, FILENAME, PHOTO_EXTENSION)

      // Setup image capture metadata
      val metadata = ImageCapture.Metadata().apply {

        // Mirror image when using the front camera
        isReversedHorizontal = lensFacing == CameraSelector.LENS_FACING_FRONT
      }

      // Create output options object which contains file + metadata
      val outputOptions = ImageCapture.OutputFileOptions.Builder(photoFile)
          .setMetadata(metadata)
          .build()

      // Setup image capture listener which is triggered after photo has been taken
      imageCapture.takePicture(outputOptions, cameraExecutor, object : ImageCapture.OnImageSavedCallback {
        override fun onError(exc: ImageCaptureException) {
          Timber.e(exc, "Photo capture failed: ${exc.message}")
        }

        override fun onImageSaved(output: ImageCapture.OutputFileResults) {
          val savedUri = output.savedUri ?: Uri.fromFile(photoFile)
          Timber.d("Photo capture succeeded: $savedUri")

          // Implicit broadcasts will be ignored for devices running API level >= 24
          // so if you only target API level 24+ you can remove this statement
          if (Build.VERSION.SDK_INT < Build.VERSION_CODES.N) {
            requireActivity()
                .sendBroadcast(Intent(android.hardware.Camera.ACTION_NEW_PICTURE, savedUri))
          }


          // If the folder selected is an external media directory, this is
          // unnecessary but otherwise other apps will not be able to access our
          // images unless we scan them using [MediaScannerConnection]
          val mimeType = MimeTypeMap.getSingleton()
              .getMimeTypeFromExtension(savedUri.toFile().extension)
          MediaScannerConnection.scanFile(
              context,
              arrayOf(savedUri.toFile().absolutePath),
              arrayOf(mimeType)
          ) { _, uri ->
            Timber.d("Image capture scanned into media store: $uri")
          }
        }
      })
    }

  }


4 Answers

You must overlay the text over the image yourself. I would suggest to use takePicture(Executor, …) that puts the Jpeg in memory; then, overlay your text using one of the libraries (not part of Android framework, neither of Jetpack), and save the result in file.

If you can compromise on image quality, you can draw the Jpeg on Bitmap canvas, and draw your text on top.

Go with this plugin https://github.com/huangyz0918/AndroidWM .Also, if you want to build your own this can help you to refer to.

The simple usage is like below

WatermarkText watermarkText = new WatermarkText(inputText)
            .setPositionX(0.5)
            .setPositionY(0.5)
            .setTextColor(Color.WHITE)
            .setTextFont(R.font.champagne)
            .setTextShadow(0.1f, 5, 5, Color.BLUE)
            .setTextAlpha(150)
            .setRotation(30)
            .setTextSize(20);

val bmFinal: Bitmap = WatermarkBuilder
                            .create(applicationContext, capturedImageBitmap)
                            .loadWatermarkText(watermarkText)
                            .watermark
                            .outputImage

// ##### Then save it #########

fun saveImage(bitmap: Bitmap, photoFile: File) {
    val output: OutputStream = FileOutputStream(photoFile)
    bitmap.compress(Bitmap.CompressFormat.JPEG, 100, output)
    output.flush()
    output.close()
    Toast.makeText(
        this@MainActivity,
        "Imaged Saved at ${photoFile.absolutePath}",
        Toast.LENGTH_LONG
    ).show()
}

val photoFile = File(
            outputDirectory,
            SimpleDateFormat(
                FILENAME_FORMAT, Locale.US
            ).format(System.currentTimeMillis()) + ".jpg"
        )

saveImage(bmFinal, photoFile)

May be you can use CameraSource class and put your preview/overlay inside :

val cameraSource = CameraSource.Builder(requireContext(),FakeDetector()).build()
cameraSource.start(your_preview_overlay) 

And after you have an API :

takePicture(CameraSource.ShutterCallback shutter, CameraSource.PictureCallback jpeg) 

Camera source (https://developers.google.com/android/reference/com/google/android/gms/vision/CameraSource) is for detection, but you can create a fake Detector ( nothing to detect).

@alexcohn's answer is the preferred one if you cannot afford to lose quality. However, if quality is not big deal then you can do this.

<FrameLayout
            android:id="@+id/camera_wrapper"
            android:layout_width="match_parent"
            android:layout_height="0dp"
            app:layout_constraintTop_toTopOf="@id/space1"
            app:layout_constraintBottom_toBottomOf="@id/space">

            <androidx.camera.view.PreviewView
                android:id="@+id/camera_view"
                android:layout_width="match_parent"
                android:layout_height="match_parent" />

            <ImageView
                android:id="@+id/selfie"
                android:layout_width="match_parent"
                android:layout_height="match_parent"
                android:scaleType="centerCrop"
                android:visibility="gone"
                tools:visibility="visible"
                tools:background="@color/gray" />

            <ImageView
                android:id="@+id/overlay"
                android:layout_width="match_parent"
                android:layout_height="match_parent"
                android:scaleType="centerCrop"
                tools:src="@drawable/full_frame_gd" />

        </FrameLayout>

PreviewView has a build in function that gives you bitmap of the preview

val bitmap = binding.cameraView.bitmap
binding.selfie.setImageBitmap(bitmap)
binding.selfie.visibility = View.VISIBLE
cameraExecutor.shutdown()
binding.cameraView.visibility = View.GONE

Now you have two images view one for selfie and one for overlay. You can't take screen shot of the previewView. There are some limitations to it that I'm not too sure of. but, I'm sure there might be a way around it.

From here you can just take a screen capture of the two combined image views like this

private fun captureView(view: View, window: Window, bitmapCallback: (Bitmap)->Unit) {
    if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
      // Above Android O, use PixelCopy
      val bitmap = Bitmap.createBitmap(view.width, view.height, Bitmap.Config.ARGB_8888)
      val location = IntArray(2)
      view.getLocationInWindow(location)
      PixelCopy.request(
          window,
          Rect(
              location[0],
              location[1],
              location[0] + view.width,
              location[1] + view.height
          ),
          bitmap,
          {
            if (it == PixelCopy.SUCCESS) {
              bitmapCallback.invoke(bitmap)
            }
          },
          Handler(Looper.getMainLooper()) )
    } else {
      val tBitmap = Bitmap.createBitmap(
          view.width, view.height, Bitmap.Config.RGB_565
      )
      val canvas = Canvas(tBitmap)
      view.draw(canvas)
      canvas.setBitmap(null)
      bitmapCallback.invoke(tBitmap)
    }
  }

and in takePhoto() function you can remove the imageCapture.takePicture logic and replace it with this.

Handler(Looper.getMainLooper()).postDelayed({
        captureView(binding.cameraWrapper, requireActivity().window) {
          // your new bitmap with overlay is here and you can save it to file just like any other bitmaps.
        }
      }, 500)
Related