Send data from Web Worker to the main thread

user_1794179 0
Viewed 980

My application spawns a Web Worker which processes many different tasks and for the computationally intensive ones with large data sets the web worker spawns more "sub-workers" to use the whole available CPU power, i.e. the first worker and the sub-workers do the processing in equal shares. Once finished the result must be returned to the main thread using postMessage() (which works fine from the first web worker).

Main thread <--> Web Worker <--> 2-4 Sub-Workers

How do I send a message with a transferable array buffer from the sub-workers to the main thread? If I use self.postMessage() the message goes to the first "Web Worker" instead.

web-worker concurrency javascript
1 Answers

You could pass a MessageChannel to your sub-workers.
If you never used it, I invite you to read this answer of mine explaining the basics of this API.

Here is a demo where the main thread generates the MessageChannel, and then the main-worker passes the sent MessagePort to the sub-worker which is now able to communicate and transfer data directly with the main thread.

const sub_worker_url = makeWorkerURL( `
  onmessage = (evt) => {
    // grab the MessagePort
    // now we can communicate directly with main
    const port = evt.ports[ 0 ];
    
    // do some computation
    const chunk = new Uint8Array( 256 );
    crypto.getRandomValues( chunk );

    // send the buffer to main directly
    port.postMessage( chunk, [ chunk.buffer ] );
  };
` );
const main_worker_url = makeWorkerURL( `
  onmessage = (evt) => {
    const subworker = new Worker( "${ sub_worker_url }" );
    // transfer the MessagePort to the subworker
    subworker.postMessage( "", [ evt.ports[ 0 ] ]  );
 };
` );

const main_worker = new Worker( main_worker_url );
// sub-worker will talk to us directly trhough this channel
const channel = new MessageChannel();
// we listen in one of the two ports
channel.port1.onmessage = ({ data }) => console.log( "received in main", data );
// we transfer the port to the worker,
// which will itself transfer it to the sub-worker
main_worker.postMessage("", [channel.port2]);


function makeWorkerURL( content ) {
  const blob = new Blob([content], { type: "text/javascript" });
  return URL.createObjectURL( blob );
}

You could also obviously rewrite this in a way the main-worker generates the MessageChannels and passes both ports to the main and to the sub-workers:

const sub_worker_url = makeWorkerURL( `
  // same as in previous demo
  onmessage = (evt) => {
    const port = evt.ports[ 0 ];
    const chunk = new Uint8Array( 256 );
    crypto.getRandomValues( chunk );
    port.postMessage( chunk, [ chunk.buffer ] );
  };
` );
const main_worker_url = makeWorkerURL( `
  const ports = [];
  for( let i = 0; i < navigator.hardwareConcurrency - 2; i++ ) {
    const channel = new MessageChannel();
    const subworker = new Worker( "${ sub_worker_url }" );
    // transfer one of the MessagePorts to the subworker
    subworker.postMessage( i, [ channel.port1 ]  );
    ports.push( channel.port2 );
  };
  // [optional] (we could send one message per port in the loop)
  // send all the generated ports to main
  // in a single message
  self.postMessage( "", ports );
` );

const main_worker = new Worker( main_worker_url );

main_worker.onmessage = ({ ports }) => {
  ports.forEach( (port, index) => {
    port.onmessage = ({ data }) =>
      console.log( "received in main from subworker #" + index, data.slice( 0, 5 ), "..." );
  } );
};


function makeWorkerURL( content ) {
  const blob = new Blob([content], { type: "text/javascript" });
  return URL.createObjectURL( blob );
}

The design is up to you.

Now, depending on your situation and the browsers you need to support, you may also want to look into SharedArrayBuffers, which sound like more suitable for your case, since all Workers could work on the same and only buffer, but unfortunately this API isn't widely supported (basically only Blink browsers do).

user_37027970
Related