How to compose (encoded) pixels into videos / live-streams in Flutter?

Viewed 299

I am trying to make an OBS-like app using Flutter.

I was trying to use Flutter engine to draw widgets onto the video frames, along with screen, with separated layers.

I came up with a bad way, which:

  • use RenderRepaintBoundary to get images of specific area.
  • use ffmpeg to compose these .png series into video with h.264 encoding.
  • (then maybe use .mp4 files to publish as video stream??)

, which is baaad in real-time performance and efficiency apparently.

(relevant code)

// some_page.dart

int index = 0;

  Future<void> onTestPressed() async {
    int i = 0;
    while (i++ < 600) {
      try {
        var render = repaintKey.currentContext!.findRenderObject()
            as RenderRepaintBoundary;
        double dpr = window.devicePixelRatio;
        var byteData = await render
            .toImage(pixelRatio: dpr)
            .then((image) => image.toByteData(format: ImageByteFormat.png));

        var tempDir = await getTemporaryDirectory();
        var fileName = '${tempDir.path}/frame_${index++}';

        var bytes = byteData!.buffer.asUint8List();
        var file = File(fileName);
        if (!file.existsSync()) {
          file.createSync();
        }

        await file.writeAsBytes(bytes);
        // OpenFile.open(fileName);
      } catch (e) {
        if (kDebugMode) {
          print(e);
        }
      }
    }
  }

I know that Flutter uses Skia as its graphic engine, and could I use Skia ability (by drawing widgets) somehow so as to produce video frames more directly?

Thank you.

0 Answers
Related