Flutter Nested Scroll and PullRefresh

I've got widget tree that contains scrollable content beyond SafeArea (screen background in current context) and I need to implement PullRefresh functionality within that Scroll, but without pulling background image. How do I achieve that? I've tried NestedScrollView without any luck.

Part of my code

    return Scaffold(
      body: ScrollConfiguration(
          child: SingleChildScrollView(
              physics: AlwaysScrollableScrollPhysics(),
              child: Stack(children: <Widget>[
                SvgPicture.asset(
                  'assets/screen_background.svg',
                  width: MediaQuery.of(context).size.width,
                ),
                SafeArea(
                  child: Column(
                    children: <Widget>[
                      RefreshIndicator(
                          onRefresh: _refresh,
                          child: Column(
                            children: <Widget>[ // How do i implement nested scroll there?
                                // Rest of my screen
                            ],
                          )
                      )
                    ],
                  ),
                ),
              ]))),
    );