Flutter : how to perform object-detection in an isolate using TensorFlow?

Viewed 557

The problem

Details about goals

  • I am performing real-time object detection on a smartphone application, developped with Dart/Flutter, with TensorFlow 2.4.1, SSD MobilenetV2 model.
  • My goal here is to perform this with smooth phone camera movements.

Actual and expected results

  • Actually, my object detection is working well, but the image flow that my phone's camera displays on the screen is frequently interrupted, which causes small latencies which I clearly do not want in my app.
  • I am trying to supress this latency, in order to obtain smooth camera movements (just like if I was using the Camera app)

What I have tried

  • I used TensorFlow to build my own object detection model. I exported it in the TFLite format by using the "standard" way. This means I currently do not use the quantization process, thus my model's input type is float32[1, 640, 640, 3].
  • I managed to perform object detection thanks to this github project, and this is perfectly working, if we forget about the latencies
  • I found this other github project, which perform the object-detection inside an Isolate, and the result is the one I want : the camera flow is perfectly fluent. However, this project works with a quantized TensorFlow model (thus with a uint8[1, size, size, 3] input type), which I do not have, because quantization is not yet implemented in TensorFlow 2 (as far as I understood).

And so, I tried to combine the two projects to obtain the result I want, and used the Tflite.ObjectDetectionOnFrame() method from the tflite plugin inside an Isolate().

  List<dynamic> predict(CameraImage img) {
    List<dynamic> results;
    var startTime = DateTime.now().millisecondsSinceEpoch;

    if (_interpreter == null) {
      print("Interpreter not initialized");
      return null;
    }

    Tflite.detectObjectOnFrame(
      bytesList: img.planes.map((plane) {
        return plane.bytes;
      }).toList(),
      model: "SSDMobileNet",
      imageHeight: img.height,
      imageWidth: img.width,
      imageMean: 127.5,
      imageStd: 127.5,
      numResultsPerClass: 1,
      threshold: 0.4,
    ).then((recognitions) {
      int endTime = new DateTime.now().millisecondsSinceEpoch;
      print("Detection took ${endTime - startTime}");

      results = recognitions;

    });

    return results;
  }

The error I get

I appears that I cannot use the tflite plugin inside an Isolate. This is the error I get :

E/flutter (17341): [ERROR:flutter/runtime/dart_isolate.cc(882)] Unhandled exception:
E/flutter (17341): ServicesBinding.defaultBinaryMessenger was accessed before the binding was initialized.
E/flutter (17341): If you're running an application and need to access the binary messenger before `runApp()` has been called (for example, during plugin initialization), then you need to explicitly call the `WidgetsFlutterBinding.ensureInitialized()` first.
E/flutter (17341): If you're running a test, you can call the `TestWidgetsFlutterBinding.ensureInitialized()` as the first line in your test's `main()` method to initialize the binding.
E/flutter (17341): #0      defaultBinaryMessenger.<anonymous closure> (package:flutter/src/services/binary_messenger.dart:92:7)
E/flutter (17341): #1      defaultBinaryMessenger (package:flutter/src/services/binary_messenger.dart:105:4)
E/flutter (17341): #2      MethodChannel.binaryMessenger (package:flutter/src/services/platform_channel.dart:143:62)
E/flutter (17341): #3      MethodChannel._invokeMethod (package:flutter/src/services/platform_channel.dart:149:36)
E/flutter (17341): #4      MethodChannel.invokeMethod (package:flutter/src/services/platform_channel.dart:332:12)
E/flutter (17341): #5      Tflite.detectObjectOnFrame (package:tflite/tflite.dart:168:27)
E/flutter (17341): #6      Classifier.predict (package:poc_aquarium/tflite/classifier.dart:223:12)
E/flutter (17341): #7      IsolateUtils.entryPoint (package:poc_aquarium/utils/cameraModule/isolateUtils.dart:48:44)
E/flutter (17341): #8      _RootZone.runUnary (dart:async/zone.dart:1450:54)
E/flutter (17341): #9      _FutureListener.handleValue (dart:async/future_impl.dart:143:18)
E/flutter (17341): #10     Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:696:45)
E/flutter (17341): #11     Future._propagateToListeners (dart:async/future_impl.dart:725:32)
E/flutter (17341): #12     Future._complete (dart:async/future_impl.dart:519:7)
E/flutter (17341): #13     _StreamIterator._onData (dart:async/stream_impl.dart:1070:20)
E/flutter (17341): #14     _RootZone.runUnaryGuarded (dart:async/zone.dart:1384:10)
E/flutter (17341): #15     _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:357:11)
E/flutter (17341): #16     _BufferingStreamSubscription._add (dart:async/stream_impl.dart:285:7)
E/flutter (17341): #17     _SyncStreamControllerDispatch._sendData (dart:async/stream_controller.dart:808:19)
E/flutter (17341): #18     _StreamController._add (dart:async/stream_controller.dart:682:7)
E/flutter (17341): #19     _StreamController.add (dart:async/stream_controller.dart:624:5)
E/flutter (17341): #20     _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:168:12)

In the end, I tried to use isolate_handler because the documentation says :

These isolates (unlike the standard ones in Flutter) can call platform plugins.

However, I did not managed to install it, because it appears that it is incompatible with the tflite plugin

My questions :

  1. How to perform object-detection inside an Isolate, in order to obtain smooth camera movements ?
  2. Is using an Isolate the only solution for this ?
0 Answers
Related