I am making an app in flutter that uses the camera of the device to do filtering with opencv over current video preview.
- I am using the camera plugin
- With startImageStream I am obtaining the frames of the video
- I am doing the filter in C++, so I use ffi to send the information to the filter. Here is the header of the function in c++
__attribute__((visibility("default"))) __attribute__((used))
uint8_t* process_image(int32_t width, int32_t height, uint8_t *bytes)
I have a native_opencv.dart as follows
import 'dart:ffi' as ffi;
import 'dart:io';
import 'package:ffi/ffi.dart';
// C function signatures
typedef _process_image_func = ffi.Pointer<ffi.Uint8> Function(ffi.Int32 width, ffi.Int32 height, ffi.Pointer<ffi.Uint8> bytes);
// Dart function signatures
typedef _ProcessImageFunc = ffi.Pointer<ffi.Uint8> Function(int width, int height, ffi.Pointer<ffi.Uint8> bytes);
// Getting a library that holds needed symbols
ffi.DynamicLibrary _lib = Platform.isAndroid
? ffi.DynamicLibrary.open('libnative_opencv.so')
: ffi.DynamicLibrary.process();
// Looking for the functions
final _ProcessImageFunc _processImage = _lib
.lookup<ffi.NativeFunction<_process_image_func>>('process_image')
.asFunction();
ffi.Pointer<ffi.Uint8> processImage(int width, int height, ffi.Pointer<ffi.Uint8> bytes)
{
return _processImage(width, height, bytes);
}
- Here I am stuck. I need to return the frames of the video processed in the filter in C++ to the app and display it on the screen. I thought I could use CameraController and feed it with an array of bytes, but I can't figure out how to do that (if possible). This is what I have until now in main.dart for that part:
void _initializeCamera() async {
// Get list of cameras of the device
List<CameraDescription> cameras = await availableCameras();
// Create the CameraController
_camera = new CameraController(cameras[0], ResolutionPreset.veryHigh);
_camera.initialize().then((_) async{
// Start ImageStream
await _camera.startImageStream((CameraImage image) => _processCameraImage(image));
});
}
Future<void> _processCameraImage(CameraImage image) async
{
Pointer<Uint8> p = allocate(count: _savedImage.planes[0].bytes.length);
// Assign the planes data to the pointers of the image
Uint8List pointerList = p.asTypedList(_savedImage.planes[0].bytes.length);
pointerList.setRange(0, _savedImage.planes[0].bytes.length, _savedImage.planes[0].bytes);
// Get the pointer of the data returned from the function to a List
Pointer<Uint8> afterP = processImage(_savedImage.width, _savedImage.height, p);
List imgData = afterP.asTypedList((_savedImage.width * _savedImage.height));
// Generate image from the converted data
imglib.Image img = imglib.Image.fromBytes(_savedImage.height, _savedImage.width, imgData);
}
I don't know how to show the filtered frames of the preview video from the camera on the app screen.
Any help is appreciated, thank you.