I am working on a (web-) animation that moves lips with played audio speech. I use Javascript getByteFrequencyData() to determine low and high frequencies and get a very realistic mouth movement.
The code runs perfectly on all Windows browsers, on android (chrome) but I face a huge problem on IOS with Safari and also with Chrome: getByteFrequencyData always returns a constant value. No errors are logged to the console so all objects are created, the audio plays (as started in a user interaction context).
UPDATE: I tested Safari, Firefox, Edge and Chrome, all with the same result.
UPDATE2: The whole app does not work on Safari OSX but on Chrome OSX. On Chrome OSX (Catalina) getByteFrequencyData also works well. It seems to be an iOS only issue.
This is how I init the audio system (once):
...
audio = document.getElementById("audio");
audio.addEventListener('ended', onAudioPlayed, false);
var AudioContext = window.AudioContext || window.webkitAudioContext;
ctx = new AudioContext();
audioSrc = ctx.createMediaElementSource(audio);
audioSrc.connect(ctx.destination);
analyser = ctx.createAnalyser();
frequencyData = new Uint8Array(analyser.frequencyBinCount);
...
And I use the requestAnimationFrame-Method to feed the animation:
...
audio.src = "audio/" + audioQueue.shift();
console.log("play: "+audio.src);
audio.type = "audio/mp3";
analyser.fftSize = 64;
analyser.smoothingTimeConstant = 0.4;
analyser.minDecibels = -80;
analyser.maxDecibels = -20;
audioSrc.connect(analyser);
analyser.getByteFrequencyData(frequencyData);
function renderFrame() {
if (!audio.paused) requestAnimationFrame(renderFrame);
analyser.getByteFrequencyData(frequencyData);
avgHigh = 0;
avgLow = 0;
count = analyser.frequencyBinCount;
//console.log("rendermund " + frequencyData[0]);
for (i = 0; i < count; i++) {
if (i < count/4)
avgLow += frequencyData[i]
else
avgHigh += frequencyData[i];
}
console.log("rendermund " + avgHigh + "/" + avgLow);
avgLow = avgLow/2000;
avgHigh = avgHigh/2000;
if (avgLow > 1) avgLow = 1;
if (avgHigh > 1) avgHigh = 1;
mund(avgHigh,avgLow); // Here I draw the mouth...
}
audio.load();
audio.play();
renderFrame();
...
Is there an alternative way to get any live information about the samples currently played?
The strange thing is that I do not get any error messages - getByteFrequencyData() returns the array of the expected size but with constant values (0) on iOS. I tested with the latest IO Version on Ipad Air, IPad Air 2 and IPads of the 5th and 6th generation all with the same result.