Exclude Widget from Listener in Flutter

I have a Listener widget to detect scroll gestures on a Column of widgets. Let's say my screen looks like this:

What I'd like to achieve is that the Listener detect all gestures on all widgets except for gestures on the orange area. What is the best practice for this? How can I exclude this orange widget from the Listener region?

Here is my code:

import 'package:flutter/material.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({Key? key}) : super(key: key);

  @override
  Widget build(BuildContext context) {
    return const MaterialApp(
      title: 'Example',
      home: Foo(),
    );
  }
}

class Foo extends StatefulWidget {
  const Foo({Key? key}) : super(key: key);

  @override
  State<Foo> createState() => _FooState();
}

class _FooState extends State<Foo> {
  @override
  Widget build(BuildContext context) {
    return Listener(
      onPointerMove: (event) => print(event),
      child: Scaffold(
        body: Column(
          mainAxisAlignment: MainAxisAlignment.center,
          children: [
            Expanded(child: Container(color: Colors.green)),
            Expanded(
              child: Row(children: [
                Expanded(child: Container(color: Colors.yellow)),
                Expanded(child: Container(color: Colors.orange)),
                Expanded(child: Container(color: Colors.yellow)),
              ]),
            ),
            Expanded(child: Container(color: Colors.red)),
          ],
        ),
      ),
    );
  }
}