BottomNavigationView stops working after navigating from another fragment

I have 2 items in my BottomNavigationView:

    <item
        android:id="@+id/first_fragment"
        android:enabled="true"
        android:icon="@drawable/ic_round_home"
        android:title="@fragment_first_title" />

    <item
        android:id="@+id/second_fragment"
        android:enabled="true"
        android:icon="@drawable/ic_round_confirmation_number"
        android:title="@string/fragment_second_title" />

I have setup my BottomNavigationView to work with the NavController:

binding.bottomNavigationBar.setupWithNavController(navController)

I also happen to have a thirdFragment which is not included in the BottomNavigationView. So, the scenario is I first navigate from firstFragment to thirdFragment. Then from there, I navigate to secondFragment:

findNavController().navigate(thirdFragmentDirections.actionThirdFragmentToSecondFragment())

Now that I've land on secondFragment, selecting firstFragment in BottomNavigationView won't navigate to it! I know it has something to do with multiple backstacks feature in navigation component, but the more I search about it the more I get confused.