Android dark mode follow system returns wrong mode unless killing the app

I'm implementing dynamic dark mode change in the app I'm working on and It's working great for MODE_NIGHT_YES and for MODE_NIGHT_NO but when I'm setting this on MODE_NIGHT_FOLLOW_SYSTEM it does apply the theme but not with the right mode selected by the user in the phone's settings.

I'm setting the theme by calling

fun updateThemeSelection(mode: Int) {
    selectedMode = mode // Saving to shared preference to remember user choice
    setDefaultNightMode(mode)
}

The way we check if the system is set to dark mode or not is with this function (I tried other configurations as well)

fun Configuration.isSystemInDarkMode(): Boolean {
    val defaultNightMode = AppCompatDelegate.getDefaultNightMode()

    if (defaultNightMode == AppCompatDelegate.MODE_NIGHT_YES) return true
    if (defaultNightMode == AppCompatDelegate.MODE_NIGHT_NO) return false

    return (uiMode and Configuration.UI_MODE_NIGHT_MASK) == Configuration.UI_MODE_NIGHT_YES
}

This does work well except for MODE_NIGHT_FOLLOW_SYSTEM that only works after force closing the app in all other cases it returns the last known mode.

The weird thing is that if I set android:configChanges="uiMode" on the activity in the manifest fun onConfigurationChanged(newConfig: Configuration) is being triggered when I toggle the dark mode quick settings but when running isSystemInDarkMode() on the config I receive the last known mode until the app is killed...

I even debugged the bits of uiMode and Configuration.UI_MODE_NIGHT_MASK and they return the last values.

I also tried using setLocalNightMode(int) and applyDayNight() and I received the same result.

I don't think it's a device-specific bug because I run it both on an emulator and on a OnePlus 7 pro, both running android 11.