HttpContext.Response underlying socket shuts down unexpectedly

I have a HttpListener that listen for any file requests on the localhost (i.e 192.168.0.10/Foobar.ext) on the specified port (in this case specifically it is a HLS stream using .m3u8 header files and .ts video files. But the system should work for any type of file).

IAsyncResult result = listener.BeginGetContext(new AsyncCallback(HttpRequestListenerCallback), listener);
result.AsyncWaitHandle.WaitOne(1);

When a request is made the callback creates a HttpListenerContext for the requested file (and that file only), and extracts the file name like this:

HttpListenerContext context = listener.EndGetContext(result);
string fileName = context.Request.Url.AbsolutePath.Substring(1);

the context gets added to a dictionary called httpContexts and linked to an int commandSequenceNumber to keep track of requests.

If the Filename is valid a requests gets send to the server to download the file. The file gets downloaded and gets put into a byte array called totalBufferdata. up to here everything works perfectly.

Now I want to write the byte data of the requested (video)file back to the response (HttpListenerContext.Response) of the context on which the file was requested

To do this i use the following piece of code (this happens after the file has been completely downloaded):

HttpListenerResponse response = httpContexts[commandSequenceNumber].Response; //Get the appropriate context from the dictionary
response.ContentLength64 = totalBufferdata.Count;//set the length response body
Stream responseStream = response.OutputStream; //The stream to which the repsonse needs to be written
try
{
   currentContext.Response.OutputStream.Write(dataNoResponseHeader, 0, dataNoResponseHeader.Length);
}
catch (IOException e)
{
   Debug.LogError("Exception in writing to response::" + e);
   Debug.LogError("innerException:: " + e.InnerException);
}
currentContext.Close();//close the request and response stream

This sends the response back over the context of the request (I.e 192.168.0.10/Foobar.ext, over the same port.)

Now this works fine, aslong as there is a fast, reliable internet connection. When the internet connnection is slow, or inconsistent I start getting the exception:

System.IO.IOException: Unable to write data to the transport connection: The socket has been shut down. ---> System.Net.Sockets.SocketException: The socket has been shut down

With the inner exception being:

System.Net.Sockets.SocketException (0x80004005): The socket has been shut down

I've looked up what the HResult of 0x80004005 corrolated to on msdn, but that is just "E_FAIL Unspecified failure", so no luck there.

I've been unable to figure out why it would throw an expection of the socket closing down (and why it happens on the localhost part, but only when connectivity is bad). I make sure all the data needed is in the totalBufferData, so a low internet speed shouldn't influence this, as all data is already downloaded before i write it to the response. I made sure i do not close the context prematurely anywhere in my code either.

So far i've been unsuccesful in finding a way to get to the underlying socket of HttpListener. I also tried casting response.OutputStream to a NetworkStream, and get the socket from the NetworkStream, but that cast is not valid (Which confused me as they're both IO streams?). Thinking it may be a closing issue i also tried

using(Stream testStream = response.OutputStream)
{
    testStream.Write(totalBufferdata.ToArray(), 0, totalBufferdata.Count);
}

I feel like this issue is related to a timeout somewhere. But the Listener doesn't hit the default time-out time. According to MSDN all default time-outs should be 2 minutes. To which I get nowhere close. And i think that the exception returned in case of a time-out should be ObjectDisposedException Write(Byte[], Int32, Int32) was called after the stream was closed. as I would imagine timing out would dispose of the connection, instead of crash it? Is that a misunderstanding?

The requests and reponses are done on seperate threads. But a request that gets done while a response is still in progress gets queued up, and waits for the response to finish before it starts a new response.

Is there a way to get and debug the underlying socket to find out why it closes/crashes? Or maybe an alternative approach to requesting a file over localhost, and responding to this that doesn't use HttpListener?

Some additional information: It is for a Unity application (Unity 2019.1) using scripting runtime version .Net 4.x equivalent, with .Net 4.x api compatibility level and IL2CPP scripting backend. But neither of the classes handling the request or response inherit from monobevahiour (which isn't even possible on threads in unity). Building for Android.

_{The bounty has ended, but I will open a new one fpr whoever has some valuable information!}