Make first item (or padding) of SingleChildScrollView not scrollable and delegate the touch events

I have a Stack with two Columns.

First column contains contains only one child (1) with InteractiveViewer which has defined height (for example 250). It changes its content's zoom and translation based on scroll offset of SingleChildScrollView from second column. It shouldn't be pushed out of the screen while scrolling, so it's under the scroll view in a stack.

The second column have SingleChildScrollView with top padding (2) OR first view (3) that matches the (1) height.

Now I'd like to make the top padding (2) or view (3) not scroll the SingleChildScrollView but pass those touch events to InteractiveViewer. Doesn't matter whether the solution use padding or empty view, I just wanted to note here than what I want can be achieved with padding or view. I tried both but failed.

I tried the GestureDetector, IgnorePointer and AbsorbPointer, but seems like the SingleChildScrollView always get the touch events. I haven't found a way to make the top padding not scrollable too.

Basically I'd like to achieve something similar to the attached gif, except that I don't need the "Collapsing Header" text animation and app bar. Just pay attention to the mountains that hide below the scroll view. The scroll view should take entire screen once the scroll offset is equal padding/view height (250 in this example).

Is that possible somehow? My logic behind InteractiveViewer is way more complicated than the example provided below, I just simplified it to make the question easier to understand.

 Stack(children: [
          Column(
            children: [
              Container( // (1) Widget that should get the touch events
                height: 250,
                child: InteractiveViewer(...)
              ),  
            ],
          ),
          Column(
            children: [
              Expanded(
                child: SingleChildScrollView(
                  padding: EdgeInsets.only(top: 250), // (2) Either not scrollable padding
                  child: Column(
                    children: [
                      Container(height: 250), // (3) or not scrollable first item
                      Container(...)
                    ],
                  ),
                ),
              ),
            ],
          ),
        ]);