Flutter StreamBuilder complex widget freezes screen

The main page of our Flutter app is pretty "complex", as it is a CustomScrollView that embeds a list of horizontal ListViews, animations, and other widgets below the CustomScrollView.

We use a StreamBuilder, that displays an CircularProgressIndicator while the data is being fetched from our APIs, and then renders the whole page.

When the data is ready, we notice that there is a short delay (maybe 500ms) where the CircularProgressIndicator "freezes" while the page is being rendered.

If I replace the whole page by a simple Container, this behavior does not happen as everything is super fast to render; the indicator never never freezes.

DEMO

video.mp4 (Flutter 1.25.0-5.0.pre.140 / Debug / Physical Samsung S8)

Simplified Sample Code:

 @override
  Widget build(BuildContext context) {
    return CupertinoPageScaffold(
        child: StreamBuilder(
            stream: _myStream,
            builder: (BuildContext context, AsyncSnapshot snapshot) {
              if (snapshot.connectionState == ConnectionState.waiting) {
                return Container(
                  child: Center(
                    child: CircularProgressIndicator(
                      backgroundColor: Colors.transparent,
                      valueColor: AlwaysStoppedAnimation<Color>(Colors.blue),
                    ),
                  ),
                );
              }
              // Complex page widget that take time to draw, freezes indicator
              return _buildHomePage(data);

              // Counter-example, does not freeze because fast to draw
              return Container(color: Colors.blue);
            }
        )
    );
  }

How can I prevent this freeze behavior?