voice receiving ends for no apparent reason

I am trying to make a discord bot that is able to have a conversation. The bot joins a Voice Channel and receives voice audio from everyone else in the channel. Pycord provides the class VoiceClient for handling Receiving and Sending audio. One problem is telling the bot when to listen to the users and when to stop listen for processing the received audio and calculating a response. VoiceClient.start_recording() and VoiceClient.stop_recording() can only be controlled by commands because pycord doesn't provide an indication of when audio gets sent by the users (indication of that green border when someone is talking). Therefore I have to controll that by checking when data gets written to the sink.

And here comes my problem: First when starting the recording (VoiceClient.start_recording()) everything is fine and I can print out the received voice data on my terminal but at some point it randomly stops, which is because there is no new data getting received. The problem lies in line 806 in voice_client.py: ready, _, err = select.select([self.socket], [], [self.socket], 0.01). The VoiceClient socket stays unready and err stays None.

I am fairly new to everything and couldn't figure out why this happens. Whats a possible cause?

example:

import discord   
import io 
from discord.sinks import WaveSink as PycordWaveSink, Filters, AudioData
 
bot =  discord.Bot() 

class WaveSink(PycordWaveSink): 
    
    @Filters.container  
    def write(self, data, user):  
        if user not in self.audio_data:
            file = io.BytesIO()
            self.audio_data.update({user: AudioData(file)})

        file = self.audio_data[user] 
        file.write(data)     
        print(data) 


@bot.slash_command(guild_ids=[GUILD_ID])
async def join(ctx):  
    await ctx.author.voice.channel.connect()  
    await ctx.respond(f'listening to {ctx.author.name}', ephemeral=True) 
    bot.voice_clients[0].start_recording(WaveSink(), lambda x: x)     


bot.run(TOKEN) 

For reconstructing the problem join a voice channel together with the bot and start to speak until the voice data doesn't get printed anymore