Flutter - RefreshIndicator on a child of CustomScrollView

So, I have a CustomScrollView with a SliverAppBar and a SliverList. The SliverList has multiple non-sliver children widgets.

CustomScrollView(
  slivers: [
    SliverAppBar(title: Text('some title')),
    SliverList(
      delegate: SliverChildListDelegate(
        [
          // non sliver children
        ],
      ),
    ),
  ],
);

I want the content (non sliver children) of the CustomScrollView to have pull down to refresh. I don't want to put the RefreshIndicator on top of the scroll view because that will also reload the SliverAppBar.

I tried wrapping all the non-sliver children with a Column and wrapping that Column in the RefreshIndicator but that didn't work.

How do I do this? Thanks.