Animating HTML Video with requestAnimationFrame

user_4157821 0
Viewed 2165

I would like to use requestAnimationFrame to play an HTML <video> element. This is useful because it offers greater control over the playback (e.g. can play certain sections, control the speed, etc). However, I'm running into an issue with the following approach:

function playAnimation() {
  window.cancelAnimationFrame(animationFrame);
  var duration = video.seekable.end(0);
  var start = null;

  var step = function(timestamp) {
    if (!start) start = timestamp;
    const progress = timestamp - start;
    const time = progress / 1000;

    video.currentTime = time;
    console.log(video.currentTime);

    if (time > duration) {
      start = null;
    }
    animationFrame = window.requestAnimationFrame(step);
  }

  animationFrame = window.requestAnimationFrame(step);
}

In Google Chrome, the video plays a little bit but then freezes. In Firefox it freezes even more. The console shows that the video's currentTime is being updated as expected, but it's not rendering the new time. Additionally, in the instances when the video is frozen, the ontimeupdate event does not fire, even though the currentTime is being updated.

A simple demo can be found here: https://codepen.io/TGordon18/pen/bGVQaXM

Any idea what's breaking?

Update: Interestingly, controlling/throttling the animationFrame actually helps in Firefox. 

setTimeout(() => {
  animationFrame = window.requestAnimationFrame(step);
}, 1000 / FPS);

This doesn't seem like the right approach though

requestanimationframe html5-video animation html javascript
1 Answers

The seeking of the video is usually slower than one frame of requestAnimationFrame. One ideal frame of requestAnimationFrame is about 16.6ms (60 FPS), but the duration of the seek depends on how the video is encoded and where in the video you want to seek. When in step function you set video.currentTime and then do the same thing in the next frame, the previous seek operation most likely has not finished yet. As you continue calling video.currentTime over and over again, browser still tries to execute old tasks until the point it starts freezing because it is overwhelmed with the number of tasks. It might also influence how it fires the events like timeupdate.

The solution might be to explicitly wait for the seek to finish and only after that asking for the next animation frame.

video.onseeked = () => {
   window.requestAnimationFrame(step);    
}

https://codepen.io/mradionov/pen/vYNvyym?editors=0010

Nevertheless you most likely won't be able to achieve the same smooth real-time playback like in the video tag, because of how the seeking operation works. Unless you are willing to drop some current frames when the previous frame is still not ready.

Basically storing an entire image for each video frame is very expensive. One of the core video compression techniques is to store full video frame only in some intervals, like every 1 second (they are called key-frames of I-frames). The rest of the frames in between will store the difference from the previous frame (P-frames), which is pretty small compared to entire image. When video plays as usual, it already has previous frame in buffer, the only thing it needs to do is apply the difference for the next frame. But when you make a seek operation, there is no previous frame to calculate the difference from. Video decoder has to find the nearest key-frame with the full image and then apply the difference for all of the following frames up until the point it finally reaches the frame you wanted to seek to.

If you use my suggestion to wait for previous seek operation to complete before requesting for the next seek, you will see that video starts smooth, but when it gets closer to 2.5 seconds it will stutter more in more, until it reaches 2.5s+ and becomes smooth again. But then again it will start stuttering up to the point of 5s, and become smooth again after 5s+. That's because key-frame interval for this video is 2.5 seconds and the farther the timestamp you want to seek to from the key-frame, the longer it will take, because more frames need to be decoded.

user_15736380
Related