Flutter NestedScrollView didn't show floating SliverAppBar instantly?

I tried to create a scrollview with floating SliverAppBar with NestedScrollView and add content inside CustomScrollView, i need to have it separated because i also used pull_to_refresh plugin to handle api request.

But if i scroll up to show floating SliverAppBar it didn't work like what i expected, it only shown after user stop scrolling

Is there a way to achieve this with NestedScrollView? or maybe other widget, i need to put it separated from CustomScrollView because if i use floating SliverAppBar inside CustomScrollView, my loading indicator will be overlapped with floating SliverAppBar

this is the code i tried

Scaffold(
        body: SafeArea(
          child: Scrollbar(
            child: NestedScrollView(
                    headerSliverBuilder: (BuildContext context, bool innerBoxIsScrolled) {
                        // These are the slivers that show up in the "outer" scroll view.
                        return <Widget>[
                            SliverAppBar(
                                title: const Text('Books Nested'), // This is the title in the app b
                                forceElevated: innerBoxIsScrolled,
                                floating: true,
                                snap: true,
                            ),
                        ];
                    },
                    body: CustomScrollView(
                        slivers: <Widget>[
                            SliverList(
                                delegate: SliverChildBuilderDelegate(
                                        (BuildContext context, int index) {
                                        return Container(
                                            margin: EdgeInsets.all(10.0),
                                            color: Colors.grey,
                                            constraints: BoxConstraints(
                                                minWidth: 100.0,
                                                maxHeight: 160.0
                                            ),
                                        );
                                    },
                                    childCount: 30
                                ),
                            ),
                        ],
                    )
                ),
          )
        ),
    );

and the result : SliverAppBar inside NestedScrollView

i want to get something like this, this one i made with only CustomScrollView to store SliverAppBar and SliverList.

SliverAppBar inside CustomScrollView