Rebuilding nested navigator causes focus loss in top navigator

My app is build with a MaterialApp.router() as the outmost widget.

The routerDelegate is a navigator, which is the top navigator. The pages of this navigator is switching between a small and a large scaffold based on MediaQuery.of(context).size.

The inner navigator has a lot of pages, these pages can't all be const as some of them require a parameter, it could be a specific id for that page so it knows what info to show. Also when accessing the page via an URL the id can be specified there. So as I see it I have no way of making sure all pages are const.

The problem is, that because I use MediaQuery.of() around the inner navigator, it naturally causes a lot of rebuilds. So for example, having a TextField in a Drawer on the top navigator, or just anywhere above the inner navigator, that TextField will lose focus when the keyboard appears because the inner navigator pages rebuilds because they are not const.

How do I set this up correctly? Should I not use MediaQuery.of() for this? Can I set up my inner pages differently?

The TextField in the middle is from the HomePage inside the inner navigator, the TextField at the bottom is in the top navigator, and the drawer is also in the top navigator.

The top navigator:

Navigator(
  key: navigatorKey,
  pages: <Page<dynamic>>[
    MaterialPage<Page<dynamic>>(
      child: MediaQuery.of(context).size.width < 400
          ? SmallScaffold(innerNavigatorKey: innerNavigatorKey)
          : LargeScaffold(innerNavigatorKey: innerNavigatorKey),
    )
  ],
  onPopPage: (Route<dynamic> route, dynamic result) {
    if (!route.didPop(result)) {
      return false;
    }
    notifyListeners();
    return true;
  },
)

The SmallScaffold:

Scaffold(
  appBar: AppBar(),
  drawer: const Drawer(child: Center(child: TextField())), // This textfield loses focus immediately!
  body: Column(
    children: [
      Expanded(
        child: Router<dynamic>(
          routerDelegate:
              InnerRouterDelegate(navigatorKey: innerNavigatorKey),
        ),
      ),
      const Divider(),
      const SizedBox(
        width: 200,
        height: 50,
        child: TextField(), // This textfield loses focus immediately!
      ),
      const SizedBox(height: 20)
    ],
  ),
  drawerEnableOpenDragGesture: false,
  resizeToAvoidBottomInset: false,
  extendBodyBehindAppBar: true,
)

The inner navigator:

Navigator(
  key: navigatorKey,
  pages: [
    // This works!
    // const MaterialPage(child: Home())
    
    // This does NOT work as the MaterialPage isn't const
    // MaterialPage(
    //   child: const Home(),
    //   key: ValueKey<MyPage>(HomePage()),
    // )
    
    // This does NOT work as the MaterialPage isn't const
    // I know there is no parameter for the Home() page here, but that is just to keep it simple! That is why I haven't made the Home() or MaterialPage() const as that would not be possible!
    MaterialPage(child: Home()) /
  ],
  onPopPage: (Route<dynamic> route, dynamic result) {
    if (!route.didPop(result)) {
      return false;
    }

    return true;
  },
)

Feel free to get my minimal project here: https://github.com/Keetz/nested-navigator-error