Navigation component implicit deep link opens the screen with an empty backstack

I created an implicit deeplink within a navigation graph. The project contains multiple activities, so it's not all just one activity and a navigation graph. The navigation graph is contained in a separate activity. I start it by passing the Uri to it, using Intent.setData(), and then when it starts, I handle it the following way:

   override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)

        ...

        if (savedInstanceState == null) {
            val navHostFragment = supportFragmentManager.findFragmentById(R.id.my_nav) as NavHostFragment
            val navController = navHostFragment.navController

            // See the update. This is no longer used
            navController.handleDeepLink(intent) 
        }
    }

The structure of the graph is basically like this:

• Main graph
  • Destination A (main graph start destination)
  • Nested graph 1
    • Destination B (nested graph 1 start destination)
    • Destination C
  • Nested graph 2
    • Destination D
    • Destination E

The deeplink navigates to Destination B. It seems to be working properly with one exception - when I navigate to Destination B using the deep link, there is nothing in the back-stack. So if I open Destination B using the deeplink, when I press back, the activity closes. I'd expect for it to navigate to Destination A.

I read some articles and other answers, which suggested to nest the graphs to avoid this (this is why I have nested graphs), however it doesn't seem to be working. What am I missing?

UPDATE I removed the explicit deep link handling in the activity. It turns out the nav component handles it by itself. However the issue remains and navigating back does not bring the user to the previous screen.

The nav graph within the activity is specified in the manifest using <nav-graph /> tag, nested in the activity declaration tag.

<activity
    android:name=".MyActivity"
    android:screenOrientation="portrait">

    <nav-graph android:value="@navigation/my_nav" />

</activity>