Solving async producer-consumer problem in JS

user_1525061 0
Viewed 4769

I have a producer of data, and a consumer of data. The producer produces asynchronously, and in turn I would like the consumer to consume asynchronously when there is data to consume.

My immediate thought to solve this problem is to use some queue object that has an awaitable shift/get, much like this async queue in the python standard

However, I searched and I couldn't find any JS libraries that have this type of data structure for me to use. I would have thought this would be a common pattern.

What is the common pattern for solving this problem in JS, and are there any libraries to help?

node.js ecmascript-6 javascript
3 Answers

If the producer of the data is just spontaneously producing data and the consumer just wants to know when there's some new data, then this sounds like the consumer should just subscribe to an event that will be triggered any time there is new data. You can just use the EventEmitter object in node.js to create an emitter that the consumer can listen to and the producer will trigger and event whenever there's new data. No external library is needed to implement this as the built-in EventEmitter object has all the tools you need to register for notifications and trigger notifications.

If the consumer of the data requests data and the producer then goes and gets it asynchronously, then this is just a typical asynchronous API. The API should probably return a promise and the producer will resolve the promise with the new data when it's ready or reject it if there was an error retrieving the data.

With the little bit of description you've provided, I don't see any particular need for an elaborate queuing system. It just sounds like publish/subscribe or a simple event notification system. If the problem is more complicated, then please give us more details on the producer of the data so we can better match the tools available in node.js to the needs of your particular problem.

user_8166200

In case of small simple program, I would just simply write something like this.

var data = [];

function Consumer()
{
    this.isConsuming = false;

    this.notify = function(){
        if(this.isConsuming)
        {
            return;
        }
        this.consumeNext();
    }

    this.consumeNext = async function(){
        this.isConsuming = true;
        if(data.length > 0)
        {
            //consume one datum
            console.log(await this.consume(data.shift()));

            //consume next datum
            this.consumeNext();
        }
        else
        {
            this.isConsuming = false;
        }
    }

    this.consume = async function(datum){
        return datum * datum;
    }
}



var consumer = new Consumer();
//call consumer.notify() when your producer produces
data.push(1,2,3,4,5);
consumer.notify();

user_103176840

This will give you another idea. In my scenario the producer creates data every 1000 milliseconds and consumer waits until producer created new data and resolved its promise.

let dataArray = []
let consumerResolver = null

function producer() {
    setInterval(() => {
        const newData = "my new Data"
        dataArray.push(newData)

        if (consumerResolver) {
            consumerResolver()
        }
    }, 1000);
}

async function consumer() {
    while (true) {
        if (dataArray.length === 0) {
            const producerPromise = new Promise((resolve) => {
                consumerResolver = resolve
            })
            await producerPromise
        }

        consumerResolver = null
        const data = dataArray.shift()
        console.log(data)
    }
}
user_23948370
Related