How do I represent an audio file for use in a neural network?

user_8011482 0
Viewed 1538

I want to create a basic convolutional autoencoder in Keras (tensorflow, python) for use on audio (MP3, WAV, etc.) files.

Basically, here's what I'm doing:

1) convert an mp3 into an array

    def mp3_to_array(original_mp3):
        blah blah blah
        return original_array

2) run array through autoencoder, output a similar (but lossy, because of the autoencoder operations) array

    def autoencoder(original_array):
        autoencoder stuff
        return new_array

3) convert an array into an mp3

    def array_to_mp3(new_array):
        halb halb halb
        return new_mp3

I know that Mel Spectrograms and Mel-frequency cepstral coefficients (mfcc's) are commonly used in classification systems. As far as I know, I can't use these, because they can't be converted back to mp3's without significant loss.

Is there an array-based, lossless * (or nearly lossless), representational conversion method that's suitable for use in a convolutional neural network, to convert an mp3 to array, and vice versa?

EDIT: Specifically, I'm asking about steps 1 and 3. I'm aware step 2 will be inherently lossy.

Thanks in advance!

conv-neural-network keras neural-network audio python
1 Answers

I would say this is less of a question about raw audio representation and more a question of is there a lossless convolutional transformation to which I would say no

... as an aside there are plenty of transformations which are lossless (or nearly so) for example when you send audio into a Fourier Transform to convert it from the time domain into its frequency domain representation then perform a second transformation by sending the freq domain representation into an inverse Fourier Transform you will now have normal time domain audio which will match your original source input audio to an arbitrary level of precision ... I know this after writing a golang project which is given an input greyscale photograph which is parsed to synthesize the per pixel light intensity level information into a single channel audio signal (inverse Fourier Transform) which is then listened to (Fourier Transform) to synthesize an output photo which matches the input photo

If you are concerned with bit level accuracy (lossless) you should avoid using mp3 and use a lossless codec or just for starters use WAV format ... any audio CD uses WAV which is just the audio curve in PCM ... its just the points on the audio curve (samples for both channels) ... in your above step 2) if you just feed the audio curve directly into your neural net it will be given your lossless audio data ... the point of typical autoencoders is by definition a lossy transformation since it throws away bit level information

There are several challenges when using audio as input into a Neural Network

1) audio has the aspect of time so depending on what you need you may want to bulk up chunks of audio samples (to make a series of windows of samples) and feed each window as a unit of data into the NN or maybe not

2) As with images, audio has a massive number of data points ... that is each point on the raw audio curve was sampled upstream and now you have typically 44,100 samples per channel per second where semantic meaning is often the result of groupings of these samples ... for instance one spoken word is an aggregate notion easily involving thousands and possibly 10 of thousands of audio sample data points ... so its critical to properly create these windows of audio samples ... bundled into creation of a window of samples is design decision of how the next window will be created : does the next window contain some samples from previous window or are all the samples new ... is the number of audio samples in each window the same or does it vary

So open up the input audio file and read it into a buffer ... to confirm this buffer was created OK just write it out to a file then play back that file and verify its playing OK ... use free open source audio tool called Audacity to open up an audio file and view its audio curve

user_1471750
Related