Does Python SpeechRecognition work with WebRTC as AudioSource?

Viewed 44

I need to implement continuous real-time speech to text that can use WebRTC as an audio source. I would love to use the speech_recognition library (here) because it has this wonderful .listen() method that works perfectly as a VAD and also makes it easy to create a wav file later to feed it to my STT model of choice. In short, I would like to structure it like this (credits to Ashutosh Dongare):

audio_input_file = "micAudioInput.wav"  
audio_output_file = "ttsAudioOutput.wav"

while True:

    try:
        with sr.Microphone() as SR_AudioSource:
            print("Say something...")
            mic_audio = r_audio.listen(SR_AudioSource,1,4) # timeout=1, phrase_time_limit=4

    except: #This is required if there is no Audio input or some error
        print("Found not capture mic input or no audio...")
        continue #continue next mic input loop if there was any error

    with open(audio_input_file, "wb") as file:
        file.write(mic_audio.get_wav_data())
        file.flush()
        file.close()

    # Check if there is input audio file saved otherwise continue listening 
    if not os.path.exists(audio_input_file):
        print("no input file exists")
        continue

    # Read audio file into batches and create input pipelie for STT
    batches = split_into_batches(glob(audio_input_file), batch_size=10)
    readbatchout = read_batch(batches[0])
    input = prepare_model_input(read_batch(batches[0]), device=device)

    #feed to STT model and get the text output
    output = stt_model(input)
    you_said = decoder(output[0].cpu())
    print(you_said)

    if(you_said == ""):
        print("No speech recognized...")
        continue
    
    #check if user wants to stop
    if(re.search("exit",you_said) or re.search("stop",you_said) or re.search("quit",you_said)):
        break

It works perfectly using a physical microphone locally. The only issue is that I am pretty sure that speech_recognition only accepts microphones of the device it is run on as an AudioSource (and I know that PyAudio only works that way) as the documentation states that the Microphone class

[c]reates a new Microphone instance, which represents a physical microphone on the computer.

Does this mean it is not possible to use it with WebRTC as source? And if not, what would be a good substitute to its .listen()? Also: Is this concept of an infinite loop generally even feasible in the browser or is it a bad idea? I plan on implementing this in Django and handling the WebSocket server with Django Channels.

0 Answers
Related