MediaSessionCompat behaviour changed in Android 12 and setPlaybackToRemote stopped working properly

Prior to Android 12, I used code below to detect physical volume buttons pressed to show my custom UI, but it stopped working on Android 12 devices and onAdjustVolume is never called when I press volume buttons:

        mediaSessionCompat = MediaSessionCompat(context, "My App")
    mediaSessionCompat?.isActive = true
    mediaSessionCompat?.setPlaybackState(
        PlaybackStateCompat.Builder()
        .setState(PlaybackStateCompat.STATE_PLAYING, PlaybackStateCompat.PLAYBACK_POSITION_UNKNOWN, 1f)
        .build())
    mediaSessionCompat?.setPlaybackToRemote(object :
        VolumeProviderCompat(VOLUME_CONTROL_ABSOLUTE, 7, 4) {
        override fun onAdjustVolume(direction: Int) {
            Log.v("My App", "Volume adjusted")
        }
    })