ModelBottomSheet does not hide on clicking scrim

I am working on building a bottom modal sheet. The problem is the sheet does not hide on clicking the scrim. The sheet is pretty straightforward and barebones, I have also tried using a modal bottom sheet with all the defaults and the same thing happens.

I was unable to reproduce the same issue outside of the project I am working on. I placed a bunch of breakpoints in an attempt to figure out what might be going wrong and found onDismiss() inside scrim(in ModalBottomSheet.kt) was not being called, which might be because the tap gesture is not getting detected I guess.

val dismissModifier = if (visible) {
    Modifier
        .pointerInput(onDismiss) { detectTapGestures { onDismiss() } }
        .semantics(mergeDescendants = true) {
            contentDescription = closeSheet
            onClick { onDismiss(); true }
        }
}