Rendering a Canvas to an Image in a way that doesn't lock up Flutter UI

user_246337 0
Viewed 663

I need to draw some images in Flutter using geometric primitives, to both show in-app and cache for later use. What I'm doing right now is something similar to this:

import 'dart:ui';

final imageDimension = 600;
final recorder = PictureRecorder();
final canvas = Canvas(
        recorder,
        Rect.fromPoints(const Offset(0.0, 0.0),
            Offset(imageDimension.toDouble(), imageDimension.toDouble())));

/// tens of thousands of canvas operations here:
// canvas.drawCircle(...);
// canvas.drawLine(...);
final picture = recorder.endRecording();

// the following call can take ~10s
final image = await picture.toImage(imageDimension, imageDimension);
final dataBytes = await image.toByteData(format: ImageByteFormat.png);

Here's an example of the outcome:

rendered canvas

I know the image operations in this amount are heavy and I don't mind them taking some time. The problem is since they're CPU bound, they lock up the UI (even though they are async and outside of any widget build methods). There doesn't seem to be any way to break the picture.toImage() call into smaller batches to make the UI more responsive.

My question is: Is there a way in Flutter to render a complex image built from geometric primitives in a way that doesn't impact the UI responsiveness?

My first idea was to do the heavy calculation inside a compute() isolate, but that won't work since the calculations use some native code:

E/flutter (20500): [ERROR:flutter/lib/ui/ui_dart_state.cc(209)] Unhandled Exception: Invalid argument(s): Illegal argument in isolate message : (object extends NativeWrapper - Library:'dart:ui' Class: Picture)
E/flutter (20500): #0      spawnFunction (dart:_internal-patch/internal_patch.dart:190:54)
E/flutter (20500): #1      Isolate.spawn (dart:isolate-patch/isolate_patch.dart:362:7)
E/flutter (20500): #2      compute (package:flutter/src/foundation/_isolates_io.dart:22:41)
E/flutter (20500): #3      PatternRenderer.renderImage (package:bastono/services/pattern_renderer.dart:95:32)
E/flutter (20500): #4      TrackRenderService.getTrackRenderImage (package:bastono/services/track_render_service.dart:111:49)
E/flutter (20500): <asynchronous suspension>

I think there is an alternative approach with CustomPaint painting just a couple of elements every frame and after it's all done screenshot it somehow using RenderRepaintBoundary.toImage(), but there's a couple of problems with this approach:

  • It's much more complicated and relies on using heuristics to choose the amount of elements that can be rendered to the canvas per frame (I think?)
  • I think the Widget would need to be visible in order for it to get rendered so I couldn't really use it to render images in the background (?)
  • I'm not exactly sure how I'd get RenderRepaintBoundary for the widget I use for rendering.

Edit: it seems like the screenshot package allows for taking screenshots that are not rendered on the screen. I don't know about its performance characteristics yet, but it seems like it could work together with the CustomPaint class. It still feels like a very convoluted workaround though, I'd be happy to see other options.

dart flutter canvas
3 Answers

The solution I'm going with for now is based on pskink's suggestion - periodically rendering the canvas to an image and bootstrapping the new canvas with the rendered image for the next batch of operations.

Since the graphics operations take in the order of ~100ms I decided not to go with scheduleTask, but delaying the processing after every batch to give the other tasks a chance to execute. It is a bit hacky, but should be good enough for now.

Here's a simplified code of the solution:

import 'package:flutter/material.dart';
import 'package:dartx/dartx.dart';
import 'dart:ui' as dart_ui;

  Future<List<Offset>> fetchWaypoints() async {
    // some logic here
  }

  Future<Uint8List> renderPreview(String imageName) async {
    const imageDimension = 1080;
    dart_ui.PictureRecorder recorder = dart_ui.PictureRecorder();
    Canvas canvas = Canvas(recorder);
    final paint = Paint()..color = Colors.black;

    canvas.drawCircle(Offset.zero, 42.0, paint);

    final waypoints = await fetchWaypoints();
    dart_ui.Image intermediateImage =
        await recorder.endRecording().toImage(imageDimension, imageDimension);

    for (List<Offset> chunk in waypoints.chunked(100)) {
      recorder = dart_ui.PictureRecorder();
      canvas = Canvas(recorder);
      canvas.drawImage(intermediateImage, Offset.zero, Paint());

      for (Offset offset in chunk) {
        canvas.drawCircle(offset, 1.0, paint);
      }

      intermediateImage =
          await recorder.endRecording().toImage(imageDimension, imageDimension);

      // give the other tasks a chance to execute
      await Future.delayed(Duration.zero);
    }
    final byteData =
        await intermediateImage.toByteData(format: dart_ui.ImageByteFormat.png);
    return Uint8List.view(byteData!.buffer);
  }
user_2463370

Probably the best approach is moving rendering from dart:ui to a library that can be used in the compute isolate.

The image package is written in pure Dart, so it should work well and it does support the primitive operations I need.

I'm currently using the Canvas solution from the other answer, and it works for the most part, but I will switch to this soon.

user_2463370

I thought the problem was you tried to deliver Picture object from Isolate to the main thread.

If what you try to do in the end is to make a ByteData, you can deliver it between Isolates. I tried making a simple snippets and it worked on DartPad.

import 'dart:typed_data';
import 'dart:ui';

import 'package:flutter/foundation.dart';

void main() async{
  final result = await compute(drawImage, null);
}

Future<ByteData?> drawImage(String? params) async {
  const imageDimension = 600;
  final recorder = PictureRecorder();
  final canvas = Canvas(
      recorder,
      Rect.fromPoints(const Offset(0.0, 0.0),
          Offset(imageDimension.toDouble(), imageDimension.toDouble())));
  // Do your heavy computations with Canvas
  // canvas.drawCircle(...);
  // canvas.drawLine(...);
  final picture = recorder.endRecording();
  final image = await picture.toImage(imageDimension, imageDimension);
  final dataBytes = await image.toByteData(format: ImageByteFormat.png);
  return dataBytes;
}

I hope it'd work for you.

user_98356550
Related