Putting the audio from a <video/> tag into an arrayBuffer to draw frequency over time

Given an element <video src="/test.mkv"/> I would like to visualize the frequency level over time.

To this end I'd like the data into an array buffer, where I can add iterate through each value and draw it on a <canvas/>.

I can get an arrayBuffer of a video with new Response(videoElement.srcObject).arrayBuffer()

I can use AudioBuffer as a source node, but not as a destination node.

I can also try and sample the video every N milliseconds with an analyzer, though this will result in gaps (Or overlapping in the data)

const analyser = ctx.createAnalyser();
analyser.fftSize = 2048;
const bufferLength = analyser.frequencyBinCount;
let dataArray = new Uint8Array(0);

setInterval(() => {
  const newData = new Uint8Array(bufferLength);
  analyser.getByteTimeDomainData(newData);
  dataArray = mergeUint8Array(dataArray, newData)
}, 100)

Is there an easier way to put the audio data from a <video/> tag inside of an arraybuffer (Or otherwise draw its frequency over time to the canvas?)