Full Screen Immersive Mode without layout bounce

How can one toggle between immersive to non-immersive mode without layout being re-calculated and thus experience a bounce effect? Here's the relevant code that I'm using to toggle between the states:

override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)
        immersive_btn.setOnClickListener {
            toggleImmersive()
        }
    }
    val isInFullScreenImmersiveMode: Boolean
        get() = window!!.decorView.systemUiVisibility and View.SYSTEM_UI_FLAG_IMMERSIVE == View.SYSTEM_UI_FLAG_IMMERSIVE

    private fun toggleImmersive() {
        if (isInFullScreenImmersiveMode) {
            showSystemBar()
        } else {
            setFullScreenImmersiveMode()
        }
    }

    fun showSystemBar() {
        window!!.decorView.systemUiVisibility = (View.SYSTEM_UI_FLAG_LAYOUT_STABLE
                or View.SYSTEM_UI_FLAG_LAYOUT_HIDE_NAVIGATION
                or View.SYSTEM_UI_FLAG_LAYOUT_FULLSCREEN)
    }

    fun setFullScreenImmersiveMode() {
        window!!.decorView.systemUiVisibility = (View.SYSTEM_UI_FLAG_LAYOUT_STABLE
                or View.SYSTEM_UI_FLAG_LAYOUT_HIDE_NAVIGATION
                or View.SYSTEM_UI_FLAG_LAYOUT_FULLSCREEN
                or View.SYSTEM_UI_FLAG_HIDE_NAVIGATION
                or View.SYSTEM_UI_FLAG_FULLSCREEN
                or View.SYSTEM_UI_FLAG_IMMERSIVE)
    }

A video showing the bounce:

Only solution I came up with is using window?.setFlags(WindowManager.LayoutParams.FLAG_FULLSCREEN, WindowManager.LayoutParams.FLAG_FULLSCREEN) in OnCreate() but that would result in another problem: The status bar won't be shown, even when not in full screen immersive mode.

Edit: layout file:

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout
    xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity">

    <Button
        android:id="@+id/immersive_btn"
        android:layout_centerInParent="true"
        android:layout_centerHorizontal="true"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:text="Toggle Immersive" />

</RelativeLayout>