How to avoid adding a fragment to the backstack if it already exists using Navigation Architecture Component?

I am using navigation component so I'm not using a FragmentManager. Let's say I have fragments A, B.

If user navigates A -> B -> A -> B and then presses the back button I want the backstack to just be A instead of A -> B -> A. In other words don't add a Fragment if it already exists in the backstack. Is this possible without using FragmentManager?