Trying to show filtered camera video preview with opencv in Flutter

Viewed 473

I am making an app in flutter that uses the camera of the device to do filtering with opencv over current video preview.

  1. I am using the camera plugin
  2. With startImageStream I am obtaining the frames of the video
  3. I am doing the filter in C++, so I use ffi to send the information to the filter. Here is the header of the function in c++
__attribute__((visibility("default"))) __attribute__((used))
uint8_t* process_image(int32_t width, int32_t height, uint8_t *bytes)

I have a native_opencv.dart as follows

import 'dart:ffi' as ffi;
import 'dart:io';
import 'package:ffi/ffi.dart';

// C function signatures
typedef _process_image_func = ffi.Pointer<ffi.Uint8> Function(ffi.Int32 width, ffi.Int32 height, ffi.Pointer<ffi.Uint8> bytes);

// Dart function signatures
typedef _ProcessImageFunc = ffi.Pointer<ffi.Uint8> Function(int width, int height, ffi.Pointer<ffi.Uint8> bytes);

// Getting a library that holds needed symbols
ffi.DynamicLibrary _lib = Platform.isAndroid
  ? ffi.DynamicLibrary.open('libnative_opencv.so')
  : ffi.DynamicLibrary.process();

// Looking for the functions
final _ProcessImageFunc _processImage = _lib
  .lookup<ffi.NativeFunction<_process_image_func>>('process_image')
  .asFunction();

ffi.Pointer<ffi.Uint8> processImage(int width, int height, ffi.Pointer<ffi.Uint8> bytes)
{
  return _processImage(width, height, bytes);
}
  1. Here I am stuck. I need to return the frames of the video processed in the filter in C++ to the app and display it on the screen. I thought I could use CameraController and feed it with an array of bytes, but I can't figure out how to do that (if possible). This is what I have until now in main.dart for that part:
 void _initializeCamera() async {
    // Get list of cameras of the device
    List<CameraDescription> cameras = await availableCameras();

    // Create the CameraController
    _camera = new CameraController(cameras[0], ResolutionPreset.veryHigh);
    _camera.initialize().then((_) async{
      // Start ImageStream
      await _camera.startImageStream((CameraImage image) => _processCameraImage(image));
    });
  }

  Future<void> _processCameraImage(CameraImage image) async
  {
    Pointer<Uint8> p = allocate(count: _savedImage.planes[0].bytes.length);

    // Assign the planes data to the pointers of the image
    Uint8List pointerList = p.asTypedList(_savedImage.planes[0].bytes.length);
    pointerList.setRange(0, _savedImage.planes[0].bytes.length, _savedImage.planes[0].bytes);

    // Get the pointer of the data returned from the function to a List
    Pointer<Uint8> afterP = processImage(_savedImage.width, _savedImage.height, p);
    List imgData = afterP.asTypedList((_savedImage.width * _savedImage.height));

    // Generate image from the converted data
    imglib.Image img = imglib.Image.fromBytes(_savedImage.height, _savedImage.width, imgData);

  }

I don't know how to show the filtered frames of the preview video from the camera on the app screen.

Any help is appreciated, thank you.

0 Answers
Related