buffering html5 video starting at #t=n

I am setting an mp4 video as a source of an html5 <video> element from a node server. In order to avoid buffering or interruption during the playback, I am letting the user start the video only when video fires an oncanplaythrough event (which will simply add a start button to the web page).

<video id="the-video" preload="auto" playsinline='true'>
  <source src="https://whatever.com" type="video/mp4">
</video>

var video = document.getElementById('the-video')
video.addEventListener('canplaythrough', ()=>{
  showStartButton()
})

The video plays black perfectly unless I decide to change the starting point to a specific time by specifing the #t parameter.

<source src="https://whatever.com#t=30" type="video/mp4">

The video jumps correctly to the desired time offset but it keeps pausing and buffering during the playback. It looks like the canplaythrough listener does not take the time offset into consideration.

Is there any way to avoid these interruptions in such a scenario? Thank you!

UPDATE:

When appending time #t, canplaythrough is fired twice in Firefox but only once in Chrome and Safari. Still no solution though