Can't swipe ViewPager that is the content of a Scaffold with Drawer in Jetpack Compose

I have a Scaffold with drawer and a content containing a ViewPager.

Scaffold(
    drawerLayout = { MyDrawer() },
) { innerPadding ->
   // Content with ViewPager
}

When I swipe the screen to change the page of the ViewPager, it instead opens the drawer. Can I disable this behavior and only open the drawer when swiped from the left-most part of the screen?

I tried setting gesturesEnabled of the Scaffold to false, but this also disables the swipe from the side of the screen.