This is not part of the code, so indicate it with the ! operator, since it is a command for the Google Colab command line (because I am first testing this code on a notebook, and then the same code can be used on a local PC). But this first line is necessary for the code to work in Google Colab.
! pip install -q transformers
And here the code:
import librosa
import torch
#for download Wav2Vec2 from the Transformers library of Hugging Face
from transformers import Wav2Vec2ForCTC, Wav2Vec2Tokenizer
In this case, I load the tokenizer and this pre-trained model
#load model and tokenizer
tokenizer = Wav2Vec2Tokenizer.from_pretrained("facebook/wav2vec2-base-960h")
model = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-base-960h")
Here is the problem, since I am loading the complete audio file and it probably saturates the RAM memory when trying to process an .mp3 file with a duration of 14:18 mins (6,54 Mb).
I choose the 16000 Hz frequency because this facebook Wav2Vec2 model is pretrained in a 16 kHz rate.
I'm not sure if the problem that causes the RAM to saturate is the weight of the file or if it is the length of the audio file? In any case I think the solution is to cut it into parts of 5 seconds, or 10 seconds or 30 seconds each (so it would be convenient to have a function with a parameter to adjust that).
input_audio_file = "CLASE 9 HOMBRE MAQUINA.mp3"
speech, rate = librosa.load(input_audio_file, sr=16000)
Finally the audio processing process, and this is where my RAM gets clogged, so I needed some help for create some kind of loop, where if the input audio is longer than X amount of seconds then the algorithm split this original audio into n-amount of parts of X amount of seconds each of them (perhaps the last part is shorter as it is less than X seconds).
Here I tokenize the inputs (in this case it is the complete audio, but I think it would be best if they were passed from audio fragments of X seconds duration each) and I would configure the tensors that contain the information of the audios in PyTorch objects.
Here the processing process:
input_values = tokenizer(speech, return_tensors = 'pt').input_values
#Store logits (non-normalized predictions)
logits = model(input_values).logits
#Store predicted id's
predicted_ids = torch.argmax(logits, dim =-1)
#decode the audio to generate text
transcriptions = tokenizer.decode(predicted_ids[0])
#And finally I would print the text after this text-to-speech process
print(transcriptions)
Here the saturation of the RAM that caused the connection to be closed (I think if I tried this code with this audio file on my PC it would just freeze)
I think that for this question the Wav2Vec2 algorith operation could be taken as an information of the context in which the program is going to work, however what I was needing is how to fragment the audio into parts and send it part by part to the processing code (perhaps it is convenient to place it inside a function that is called inside the loop) and print what is converted to text line by line in the console.
I have been able to determine the length of a audio file in seconds with librosa methods to decide whether to split it or not
speech, rate = librosa.load(input_audio_file)
x = 10
len_data = len(speech) # holds length of the numpy array
duration_in_seconds = int( len_data / rate ) # returns duration in seconds but in floats
print(duration_in_seconds)
if(duration_in_seconds >= x):
#split the audio file in x secs audio chunks
else:
#dont split the audio file