Smooth and Efficient LipSync with Morph Targets in Three.js

user_149838 0
Viewed 421

I try to perform LipSync on a ReadyPlayer.me avatar in ThreeJS. The puppet already have Viseme Face Rigged Morph Target. I'm usings Microsoft Speech SDK to retrieve the audio and the Viseme values (timecode / ID) and Map it correctly.

Demo (the 1st part) : https://www.youtube.com/watch?v=vLbQ2arXzRk

Here is my algorithm / workflow:

  • Start playing audio
  • Reset N-1 MorphTargetInfluence to 0
  • Update MorphTargetInfluence head.morphTargetInfluences[ viseme.id ] = 0.7
  • Wait until next offset setTimeout(...)
  • On the AnimationLoop I call renderer.render( scene, camera );

Questions & Issues

  • The morphing is not smooth, how can I set smooth animation from 0 to 0.7 ?
  • Should I reset previous morphTarget to 0 ? (I assume yes)
  • In this Forum they use TWEEN but I don't think it is efficient to create an object every 50ms ? And how long should last the animation may be under 100 or 200ms it's not relevant ?

Thanks, I'm new to Three.js

viseme lipsync three.js
1 Answers

The morphing is not smooth, how can I set smooth animation from 0 to 0.7 ?

It depends on your use case. It is possible to use an animation library like GSAP or Tween.js to animate morph target influences from one value to another. However, it's potentially more performant to setup an animation clip from sequence of morph targets and then use the standard animation system of three.js to playback the clip. The clip itself can be generated via the static factory method AnimationClip.CreateFromMorphTargetSequence().

Should I reset previous morphTarget to 0 ?

A single set of morph targets represents the geometry in a specific state. You normally want to set the morphTargetInfluences entry for all morph targets to 0 which should not affect the 3D object. That also means you want to animate the "previous" morph target in your list from its current influence value back to 0.

In this Forum they use TWEEN but I don't think it is efficient to create an object every 50ms ? And how long should last the animation may be under 100 or 200ms it's not relevant ?

See my first answer. It could be a performance issue if create too many objects per frame. You potentially need no third-party animation library for your use case if the built-in animation system works for you.

user_52508470
Related