How to unblock PeerJS WebRTC data channel send function for file streaming with fs.createReadStream in Electron app

user_3792705 0
Viewed 77

I have a file sharing app (Client1 sends large file to Client2) implemented with Electron. Client1 open a large file and stream it to Client2 in data chunk with fs.createReadStream. The problem I have is that the Client2 data channel does not received any chunk until all chunks are finished sending by Client1.

BTW, it cause WebRTC connection to drop as well depending on file size , such as 200MB(and PC memory possibly, such 16GB). For smaller file, everything works.

What I suspect is that fs.createReadStream on(‘data’…) handler is a loop, and PeerJS webrtc send() function is blocked, until all chunks are sent.

Here is the main pseudo code:

  1. Electron main process open file, and File is opened with fs.createReadStream

icpMain_process.js

// Handler to process ipcRederer UI file open. It store file handle in main process.
ipcMain.handle(READ_FILE_CHUNK_OPEN, async (event, fullFileName) => {

    let fileSize = 4 x 1024;
    // Default format is uint8array, not utf8. 
    this.readStream = fs.createReadStream(fullFileName, {highWaterMark: chunkSize})

   // icpRederer process makes RPC call to ipcMain process to trigger chunk read flowing from main to UI process.
   ipcMain.on(handlerName.READ_FILE_CHUNK_NEXT, (event, arg) => {
   
     this.readStream.on('data', (chunk) => {
       this.win.webContents.send(READ_FILE_CHUNK_NEXT_RESP, chunk});
  1. This is Renderer process.

icpRederer_process.js

// Process file chunks from main process.
// All chunks are streamed here to peerjs_send_data in peeerjs_data.js, such as 10000 chunks.
ipcRenderer.on(READ_FILE_CHUNK_NEXT_RESP, (event, fileChunk) => {
    // Is there a loop in fs.createReadStream? If it’s a loop, it will block webrtc data channel?
    peerjs_send_chunk(fileChunk);
  1. PeerJS code.

// peerjs_dataconnection is PeerJS data channel.

  // Sender
  peerjs_send_chunk(fileChunk) {
     let packet = (fileChunk converts to packet)
     peerjs_send_data(packet);
  }

  peerjs_send_data(packet) {
       peerjs_dataconnection.send(packet);
  }


  // Receiver handler
  // Problem: packet is not received until all 10000 chunks are sent from sender.
  peerjs_dataconnection.on('data', (packet) => {
       dispatchpacket(packet);
  });

  dispatchpacket(packet) {


  }

I tried to use “recursive scheme” and setTimeout() to make peerjs webrtc send() to be executed in a recursive call triggered by setTimeout(), but I am not convinced it can solve it (result is uncertain).

Tried this unblocking without luck, it's still blocked.

peerjs_send_data(packet) {
    unblockTask(peerjs_dataconnection, peerjs_dataconnection.send, packet);
}


function unblockTask(context, fn, param) {
  let index = 0;
  
  function doChunk() {
    if (index == 1) {
      let ret = fn.call(context, param);
    }
    index += 1;
    setTimeout(doChunk, 1);
  } 
  
  doChunk();  
}

If open a large file in main process with fs.readFileSync() and send all file content to renderer process, and use a similar technique used unblockTask() function to break it apart as chunks, loop through the chunks to send to Client2, all works. But, it's limited by file size depending on PC memory.

What’s the reason of blocking: Client2 does not receive any packet until all chunks are sent by Client1? How to solve it?

peerjs webrtc electron node.js
1 Answers

I would try simplifying your setup in a test scenario, maybe try without streaming? I am using FileReader(), reading as arraybuffer into 60,000 byte chunks, each chunk is put into a numbered object, progress on the receive end is posted back to receiver and it's working well.

One major problem was that the chunks are not always received in the order they were sent (especially using turn and sometimes using stun), so I store the numbered chunks in an object as they are received, then when complete loop the object and push to array then concenate the array to one finished arraybuffer.

//https://gist.github.com/72lions/4528834 (for concatArrayBuffers())
const concatArrayBuffers = async function(bufs){
  var offset = 0;
  var bytes = 0;
  var bufs2=bufs.map(function(buf,total){
    bytes += buf.byteLength;
    return buf;
  });
  var buffer = new ArrayBuffer(bytes);
  var store = new Uint8Array(buffer);
  bufs2.forEach(function(buf){
    store.set(new Uint8Array(buf.buffer||buf,buf.byteOffset),offset);
    offset += buf.byteLength;
  });
  return buffer
}

I've tested with 35mb mp4 with 100% success, (I check a hash before and after receiving)

let packet = (fileChunk converts to packet)

I couldn't follow what you're doing here...maybe try just slicing an arraybuffer?

I used https://webrtc.github.io/samples/src/content/datachannel/filetransfer/ as a basis for slicing the file.

user_26770340
Related