Unpausing background music without deactivating current AVAudioSession

I'm building a Unity game for iOS and I need to, at runtime, pause any background audio (Spotify, etc.) for a short period of time, then turn it back on. I'm using an ObjectiveC plugin to handle the native behaviour.

Usually I would use AVAudioSession.setActive but turning the Unity audio session back on causes a very noticable freeze in the game, and it can't be threaded. So currently my solution is: when I need to pause it I am changing the category to SoloAmbient (which turns off the music) and then changing it back to Ambient when I'm done.

The problem is that when I set it back to Ambient the background music does not automatically unpause. I need to find a way to unpause the background music again without using AVAudioSession.setActive and AVAudioSessionSetActiveOptionNotifyOthersOnDeactivation.

How can I do this?

(void)audioFocusOn {
     AVAudioSession *audioSession = [AVAudioSession sharedInstance];
     [audioSession setCategory:AVAudioSessionCategorySoloAmbient error: nil];
});

(void)audioFocusOff {  
    AVAudioSession *audioSession = [AVAudioSession sharedInstance];
    [audioSession setCategory:AVAudioSessionCategoryAmbient error: nil];
});