I want to do some processing on audio files without playing them, just math. I doubt if I'm doing right and have several questions. I read some examples but most of them is about video streaming and there is no working with raw data at all.
I prepared an mp3 file that has 2 identical channels, i.e. it is stereo but the left and the right are the same. After decoding I expected to get buffer with pairs of equal numbers because PCM-16 stores samples of channels alternately, like {
LRLRLR...}, right? E.g.:{
105105601601-243-243-484-484...}.But I get pairs of close numbers but not equal:
{
-308-264-1628-1667-2568-2550-4396-4389}Does mp3 algorithms encode the same values differently or why?
I want to process data in packs of 1024 samples. If there will be not enough samples for another pack I want to save the rest until next batch of raw data (see
mExcessin code). Is there guarantee that order will be kept?I used to understand "sample" as every single value of audio data. Here I see
MediaExtractor::readSampleDataandMediaExtractor::advancemethods. The first returns ~2000 values, in description of the second said "Advance to the next sample". Is this just overlap of naming? I saw couple of examples where these methods are called in pair in loop. Is my usage correct?
Here is my code:
public static void foo(String filepath) throws IOException {
final int SAMPLES_PER_CHUNK = 1024;
MediaExtractor mediaExtractor = new MediaExtractor();
mediaExtractor.setDataSource(filepath);
MediaFormat mediaFormat = mediaExtractor.getTrackFormat(0);
mediaExtractor.release();
MediaCodecList mediaCodecList = new MediaCodecList(MediaCodecList.ALL_CODECS);
mediaFormat.setString(MediaFormat.KEY_FRAME_RATE, null);
String codecName = mediaCodecList.findDecoderForFormat(mediaFormat);
mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 0); // MediaCodec crashes with JNI
// error if FRAME_RATE is null
MediaCodec mediaCodec = MediaCodec.createByCodecName(codecName);
mediaCodec.setCallback(new MediaCodec.Callback() {
private MediaExtractor mExtractor;
private short[] mExcess;
@Override
public void onInputBufferAvailable(MediaCodec codec, int index) {
if (mExtractor == null) {
mExtractor = new MediaExtractor();
try {
mExtractor.setDataSource(filepath);
mExtractor.selectTrack(0);
} catch (IOException e) {
e.printStackTrace();
}
mExcess = new short[0];
}
ByteBuffer in = codec.getInputBuffer(index);
in.clear();
int sampleSize = mExtractor.readSampleData(in, 0);
if (sampleSize > 0) {
boolean isOver = !mExtractor.advance();
codec.queueInputBuffer(
index,
0,
sampleSize,
mExtractor.getSampleTime(),
isOver ? MediaCodec.BUFFER_FLAG_END_OF_STREAM : 0);
} else {
int helloAmaBreakpoint = 1;
}
}
@Override
public void onOutputBufferAvailable(
MediaCodec codec,
int index,
MediaCodec.BufferInfo info) {
ByteBuffer tmp = codec.getOutputBuffer(index);
if (tmp.limit() == 0) return;
ShortBuffer out = tmp.order(ByteOrder.nativeOrder()).asShortBuffer();
// Prepend the remainder from previous batch to the new data
short[] buf = new short[mExcess.length + out.limit()];
System.arraycopy(mExcess, 0, buf, 0, mExcess.length);
out.get(buf, mExcess.length, out.limit());
final int channelCount
= codec.getOutputFormat().getInteger(MediaFormat.KEY_CHANNEL_COUNT);
for (
int offset = 0;
offset + SAMPLES_PER_CHUNK * channelCount < buf.length;
offset += SAMPLES_PER_CHUNK * channelCount) {
double[] x = new double[SAMPLES_PER_CHUNK]; // left channel
double[] y = new double[SAMPLES_PER_CHUNK]; // right channel
switch (channelCount) {
case 1: // if 1 channel then make 2 identical arrays
for (int i = 0; i < SAMPLES_PER_CHUNK; ++i) {
x[i] = (double) buf[offset + i];
y[i] = (double) buf[offset + i];
}
break;
case 2: // if 2 channels then read values alternately
for (int i = 0; i < SAMPLES_PER_CHUNK; ++i) {
x[i] = (double) buf[offset + i * 2];
y[i] = (double) buf[offset + i * 2 + 1];
}
break;
default:
throw new IllegalStateException("No algorithm for " + channelCount + " channels");
}
/// ... some processing ... ///
}
// Save the rest until next batch of raw data
int samplesLeft = buf.length % (SAMPLES_PER_CHUNK * channelCount);
mExcess = new short[samplesLeft];
System.arraycopy(
buf,
buf.length - samplesLeft,
mExcess,
0,
samplesLeft);
codec.releaseOutputBuffer(index, false);
if ((info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) > 0) {
codec.stop();
codec.release();
mExtractor.release();
}
}
@Override
public void onError(MediaCodec codec, MediaCodec.CodecException e) {
}
@Override
public void onOutputFormatChanged(MediaCodec codec, MediaFormat format) {
}
});
mediaFormat.setInteger(MediaFormat.KEY_PCM_ENCODING, AudioFormat.ENCODING_PCM_16BIT);
mediaCodec.configure(mediaFormat, null, null, 0);
mediaCodec.start();
}
Quick code review is also welcome.