Prevent/Remove Flicker from first page load

I've been working on a generated static website with NuxtJS in where the user can select whether or not to use dark mode or just go with the default CSS media query selector. The code for that is as follows:

<template>
  <div class="container">
    <vertical-nav />
    <!-- wrapper for content; when state is updated, only then is the class added -->
    <div id="content-wrapper" :class="colorScheme !== null ? colorScheme : ''">
      <nuxt />
    </div>
  </div>
</template>
<script>
import { mapMutations, mapState } from 'vuex';
import VerticalNav from '@/components/VerticalNav.vue';

export default {
  components: {
    VerticalNav,
  },
  // Vuex state that stores ui preference, by default null
  computed: mapState(['colorScheme']),
  mounted() {
    // Vuex mutation to set initial state
    this.getPreferredColorScheme();
  },
  methods: mapMutations(['getPreferredColorScheme']),
};

I've noticed that for a first page load, the webpage will flicker/flash as shown in this screen recording. Note that this is with the cache off.

Further investigation suggests that this is only happens for when it is the first time a user visits the site as subsequent visits/refreshes don't seem to trigger this behavior. I suspect it might be related to loading in of fonts and subsequently reflows as suggested by this water fall. Please note that display=swap is not being used here and not desired. Additionally, from this profiler data from Chrome, it seems that around the flicker, the browser decides to recalculate styles and layout all the nodes in the page (relevant time from 4.26 seconds to 4.32s).

What I am asking is the flash/flicker of invisible text caused by how I am adding the optional class? If the class binding is not the issue, what is the reason and is there anything I can do be about it?

EDIT: On further inspection, this behavior seems to only occur in Chrome and might be related to the self-hosted fonts I have. Firefox and Safari don't have this repaint behavior.