Babel config file in .mjs format does not work when using the function parameter api

I am looking into converting my current babel.config.js file to a babel.config.mjs file. This should be straight forward but for some reason I am getting the following error when I try to use the function parameter api in my configuration: Cannot change caching after evaluation has completed.

Here is my config, pretty barebones:

export default function(api) {
  api.cache(true) // if I remove this then it will work
return {
  plugins,
  presets
 }
}