AVAudioPlayerNode skips a lot of sounds when stopped and started again

I'm scheduling a buffer periodically using scheduleBuffer function. The completion handler schedules the next sound when current one is done, like so:

func scheduleSounds() {
    if !isPlaying { return }
   
    while beatsScheduled < beatsToScheduleAhead {
        // Returns AVAudioTime of the next sound, based on sample time
        // first sound is always sample time 0, then increments.
        let nextTime = getNextTime()

        player.scheduleBuffer(
            soundBuffer,
            at: nextTime,
            options: AVAudioPlayerNodeBufferOptions(rawValue: 0),
            completionHandler: {
                self.queue!.sync {
                    print("Finished playing a sound!")
                    scheduleSounds()
                }
            }
        )

        beatsScheduled += 1

        if !playerStarted {
            player.play(at: nil)
            playerStarted = true
        }
    }
}

When I stop and then start the player/engine there is a delay before the sound starts playing and after logging which sounds play when I noticed that there are couple of sounds that the engine does not play. My suspicion is that the player nodes think that the sample time has already passed relative to the player node time. I think I might not be resetting or stopping something properly, here's how I start and stop the sounds:

func start() {
    do {
        try engine.start()

        isPlaying = true

        queue!.sync() {
            self.scheduleSounds()
        }
    } catch {
        print("\(error)")
    }
}

func stop() {
    isPlaying = false
        
    player.stop()
    engine.stop()
        
    playerStarted = false
}

So the problem is how do I make sure that there are no sounds that AVAudioPlayerNode skips before playing the sound?