NestJS background process which fetches data in 5 parallel actions

user_1254561 0
Viewed 48

I am working on a NestJS API which receives some article codes and saves them in a DB with status as queue and then i want to build a background process which searches for articles with queue status and fetches a website with that code and fills in the information about that article ID.

I want it to fetch codes in 5 parallel routines, each one should get a new code from the list as soon as it finished fetching the current code.

The way it's built right now, it gets 5 codes and uses Promise.all but what i don't like on this approach is that if one of the await is significantly slower than the other, they all way for the slowest one.

const promiseResult = await Promise.all(promiseList);

Which approach is the best one for me to learn in order to achieve what i am planning to do?

async-await nestjs synchronization javascript
1 Answers

Basically you do your job when all the promises are fulfilled. You are correct when you say that you wait for the slowest one, as this is exactly what await Promise.all(promiseList) does.

Now you need to ask yourself the question whether you are able to process the quicker promises while you await for the slowest one. If so, then you can iterate your promises and define a .then() for them:

for (let promise of promiseList) {
    promise.then((value) => {
        //Do something
    });
}

This way all your promises have a then handler for the case when the promise is fulfilled and your Javascript flows normally, without waiting for any of the promises. Once each promise is fulfilled, the event loop will detect that an event happened and call your callback function (in the case of the example above, the arrow function inside the then call).

Note that the then callback should never assume that the other promises were also completed, but that shouldn't be a problem in your case, because you want to work when your promise list was only partially completed as well.

user_4365600
Related