Navigation components getting back fragment result returns to start destination

I am using bottom navigationview with navigation components. Here the first menu is the start destination. Whenever I use findNavController().currentBackStackEntry?.savedStateHandle?.getLiveData<Contact>("contact") ?.observe(viewLifecycleOwner) { result -> viewModel.setContact(result) } to get result from third menu fragments child fragments fragment. Whenever I popBackstack() the child fragment it returns to the first menu ie the start destination. If I popBackstack() after commenting out the above code it returns normally to the third menu itself. Is there a way to return to the third menu itself after receiving the result from the child fragments fragment.