How to remove glitch when navigating between Fragments with different toolbars?

I am using the Jetpack Navigation Component in an app with Single Activity structure.

Since in some Fragments the toolbar needs to be different or not be there at all, I am following Google's suggestions, which basically says not to have the toolbar it in the Activity, but to move it in each fragment. So every Fragment (that needs a toolbar) has:

override fun onViewCreated(view: View, savedInstanceState: Bundle?) {
    val navController = findNavController()
    val appBarConfiguration = AppBarConfiguration(navController.graph)

    view.findViewById<Toolbar>(R.id.toolbar)
            .setupWithNavController(navController, appBarConfiguration)
}

This will set up each Toolbar for use with a NavController. And the title in the Toolbar will automatically be updated when the destination changes (assuming there is a valid label).

Now, imagine we have two fragments:

• FragmentA: it has a toolbar
• FragmentB: it doesn't have any toolbar

When I am in FragmentA the toolbar with the title "Fragment A" is displayed. When I navigate to the FragmentB (through an Action) I would expect to have no toolbar. Unfortunally for a split second the toolbar will show "Fragment B".

I think the reason is that on FragmentA the toolbar is still "listening" to the changes happening in the navigation graph (it is basically tracking the current destination) and this result in updating its own toolbar.

NOTE: For this particular scenario I could remove the label from FragmentB, since we don t need any title because there is no toolbar. But there are still some scenario where this is a problem: for example if I in FragmentA I have a standard toolbar and in FragmentB I have a prominent header like the one in the image below, I would still see a glitch.

Does anybody have idea about how this could be fixed?