I'm trying to make a simple video player that plays videos with multiple audio tracks. I want to be able to play the audio tracks at the same time. I created a simple MP4 which just says A, B, C on repeat - with each letter on a different track. That way I know which track I'm listening too, currently it just says A and I can't hear B or C. If I open the file in VLC I can hear each track so I know the file is not corrupt.
I saw that there is an experimental feature that looked promising: https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/audioTracks
However, it seems I could only enable one of the tracks at a time using the enabled flag. If multiple are enabled it would select the first - effectively no difference to what it was doing anyway. For reference here is the what the tracks look like:
audioTracks: AudioTrackList
0: AudioTrack {id: '2', kind: 'main', label: 'SoundHandler', language: 'und', enabled: true, …}
1: AudioTrack {id: '3', kind: 'main', label: 'SoundHandler', language: 'und', enabled: false, …}
2: AudioTrack {id: '4', kind: 'main', label: 'SoundHandler', language: 'und', enabled: false, …}From more searching I came across the Web Audio API. It seemed like this should be possible, but I can't find a way to extract the other tracks from the file. I tried various combinations of using a splitter node, merger node, both together. Unfortunately when I play the video I still get only the first track playing.
const audioContext = new AudioContext();
const audioSource = audioContext.createMediaElementSource(videoRef.current);
let splitter = audioContext.createChannelSplitter();
let merger = audioContext.createChannelMerger();
audioSource.connect(splitter);
splitter.connect(merger);
merger.connect(audioContext.destination);I also tried specifically setting channels:
splitter.connect(merger, 0, 0);
splitter.connect(merger, 0, 1);
splitter.connect(merger, 0, 2);In honestly I'm not sure that's the right way round, but I've tried reversing the input and outputs, and also connecting the source directly to the merger but it doesn't work. I'm guessing that the Web Audio API by default can't see the other audio tracks.
I see there's a method to get the specific track: https://developer.mozilla.org/en-US/docs/Web/API/AudioContext/createMediaStreamTrackSource
But I can't see how to get that from a media element, I could convert my video to a buffer but I would prefer not to go down this path if I don't have to. It does mention on that page it selects track order based on kind property being "audio", but it implies it uses that to simply select the first one. Even though in my files the kind is set to main it still picks up the first track.
Apologies if I'm missing something obvious. Any help will be greatly appreciated