Application stack is very busy while executing a big task, need help for micro/alternative queue

user_13454104 0
Viewed 61

our NodeJS backend application has more than 200K users and every week we do some controls/calculations based on users. The operation is similar to this:

const users = await dbService.user.findMany({}); // Array of 200K+ users
for (let index = 0; index < users.length; index++) {
  const user = users[index];
  const result = await checkUserSubscriptions({ id: user.id });
  // do operations according to result, mostly updateUser in database.
}

During the process, most of other requests are not executed or waited till this operation finishes. For example when a user tries to log in, they wait until this loop ends or they get a big delay. Because this operation does not need to be instant and can have delay or be slow, I need to use something else that does not block our main queue. What can I do to ensure this?

typescript node.js queue javascript
2 Answers

What you are trying to do is like a background job and it shouldn't be in the same process of your web service. The memory usage of procesing 200K users will impact on your web service performance. Nodejs architecture is single-threaded and It doesn't matter if you try to run 200K async tasks, your web service performance will be affected.

You can see worker_threads, useful for performing CPU-intensive JavaScript operations. There are other packages that can helps you. You can take a look at bull

user_143498080

It is a bad practice to place an await inside a loop because it renders the code synchronous and you lose the ability for parallelism: each loop iteration has to wait for the previous one to complete before being triggered.

You should use Promise.all:

const users = await dbService.user.findMany({}); // Array of 200K+ users
const results = await Promise.all(users.map((user) => {
  return checkUserSubscriptions({ id: user.id });
}))
// results is an array of returned values from checkUserSubscriptions
user_119081450
Related