The problem
Details about goals
- I am performing real-time object detection on a smartphone application, developped with Dart/Flutter, with TensorFlow 2.4.1, SSD MobilenetV2 model.
- My goal here is to perform this with smooth phone camera movements.
Actual and expected results
- Actually, my object detection is working well, but the image flow that my phone's camera displays on the screen is frequently interrupted, which causes small latencies which I clearly do not want in my app.
- I am trying to supress this latency, in order to obtain smooth camera movements (just like if I was using the Camera app)
What I have tried
- I used TensorFlow to build my own object detection model. I exported it in the TFLite format by using the "standard" way. This means I currently do not use the quantization process, thus my model's input type is float32[1, 640, 640, 3].
- I managed to perform object detection thanks to this github project, and this is perfectly working, if we forget about the latencies
- I found this other github project, which perform the object-detection inside an Isolate, and the result is the one I want : the camera flow is perfectly fluent. However, this project works with a quantized TensorFlow model (thus with a uint8[1, size, size, 3] input type), which I do not have, because quantization is not yet implemented in TensorFlow 2 (as far as I understood).
And so, I tried to combine the two projects to obtain the result I want, and used the Tflite.ObjectDetectionOnFrame() method from the tflite plugin inside an Isolate().
List<dynamic> predict(CameraImage img) {
List<dynamic> results;
var startTime = DateTime.now().millisecondsSinceEpoch;
if (_interpreter == null) {
print("Interpreter not initialized");
return null;
}
Tflite.detectObjectOnFrame(
bytesList: img.planes.map((plane) {
return plane.bytes;
}).toList(),
model: "SSDMobileNet",
imageHeight: img.height,
imageWidth: img.width,
imageMean: 127.5,
imageStd: 127.5,
numResultsPerClass: 1,
threshold: 0.4,
).then((recognitions) {
int endTime = new DateTime.now().millisecondsSinceEpoch;
print("Detection took ${endTime - startTime}");
results = recognitions;
});
return results;
}
The error I get
I appears that I cannot use the tflite plugin inside an Isolate. This is the error I get :
E/flutter (17341): [ERROR:flutter/runtime/dart_isolate.cc(882)] Unhandled exception:
E/flutter (17341): ServicesBinding.defaultBinaryMessenger was accessed before the binding was initialized.
E/flutter (17341): If you're running an application and need to access the binary messenger before `runApp()` has been called (for example, during plugin initialization), then you need to explicitly call the `WidgetsFlutterBinding.ensureInitialized()` first.
E/flutter (17341): If you're running a test, you can call the `TestWidgetsFlutterBinding.ensureInitialized()` as the first line in your test's `main()` method to initialize the binding.
E/flutter (17341): #0 defaultBinaryMessenger.<anonymous closure> (package:flutter/src/services/binary_messenger.dart:92:7)
E/flutter (17341): #1 defaultBinaryMessenger (package:flutter/src/services/binary_messenger.dart:105:4)
E/flutter (17341): #2 MethodChannel.binaryMessenger (package:flutter/src/services/platform_channel.dart:143:62)
E/flutter (17341): #3 MethodChannel._invokeMethod (package:flutter/src/services/platform_channel.dart:149:36)
E/flutter (17341): #4 MethodChannel.invokeMethod (package:flutter/src/services/platform_channel.dart:332:12)
E/flutter (17341): #5 Tflite.detectObjectOnFrame (package:tflite/tflite.dart:168:27)
E/flutter (17341): #6 Classifier.predict (package:poc_aquarium/tflite/classifier.dart:223:12)
E/flutter (17341): #7 IsolateUtils.entryPoint (package:poc_aquarium/utils/cameraModule/isolateUtils.dart:48:44)
E/flutter (17341): #8 _RootZone.runUnary (dart:async/zone.dart:1450:54)
E/flutter (17341): #9 _FutureListener.handleValue (dart:async/future_impl.dart:143:18)
E/flutter (17341): #10 Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:696:45)
E/flutter (17341): #11 Future._propagateToListeners (dart:async/future_impl.dart:725:32)
E/flutter (17341): #12 Future._complete (dart:async/future_impl.dart:519:7)
E/flutter (17341): #13 _StreamIterator._onData (dart:async/stream_impl.dart:1070:20)
E/flutter (17341): #14 _RootZone.runUnaryGuarded (dart:async/zone.dart:1384:10)
E/flutter (17341): #15 _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:357:11)
E/flutter (17341): #16 _BufferingStreamSubscription._add (dart:async/stream_impl.dart:285:7)
E/flutter (17341): #17 _SyncStreamControllerDispatch._sendData (dart:async/stream_controller.dart:808:19)
E/flutter (17341): #18 _StreamController._add (dart:async/stream_controller.dart:682:7)
E/flutter (17341): #19 _StreamController.add (dart:async/stream_controller.dart:624:5)
E/flutter (17341): #20 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:168:12)
In the end, I tried to use isolate_handler because the documentation says :
These isolates (unlike the standard ones in Flutter) can call platform plugins.
However, I did not managed to install it, because it appears that it is incompatible with the tflite plugin
My questions :
- How to perform object-detection inside an Isolate, in order to obtain smooth camera movements ?
- Is using an Isolate the only solution for this ?