SavedStateHandle in shared HILT viewmodel is always null

I have a shared Hilt ViewModel with a savedStateHandle in my constructor. I use the Android navigation component and pass the argument I need to the navigation method like so:

val direction = SomeFragmentDirections
       .actionFragmentAToFragmentB(
           someArg = foo
       )
findNavController().navigate(direction)

Now, in the receiving Fragment, when I use the Kotlin delegate by viewModles() everything works fine. But when I use by activityViewModels() the saveStateHandle is always null, however, as far as I have understood, this should still work? What is the issue here? How would you share such a viewModel, when you still want to use savedStateHandle?