How to execute long-running background functions (tasks) in production?

user3132457 0
Viewed 21

I have an app where after the user signs in with 3rd party application, the user's followings are retrieved from that service. For each of the followings, I get more data from another API and store this overall information in my database.

In my test account with 70 followings, this process takes ~20 minutes (including timeouts due to API rate limits).

const addFollowings = async followings => {
    const followingsData = []
    for (const following of followings) {
        const { id, name } = following
        const personInfo = new PersonInfo(name)
        await personInfo.init()
        const number = ...
        // ...
        followingsData.push({ name, id, number, ... })
        console.log('new following:', name)
    }
    await insertFollowingsIntoDB(followingsData)
}

This works fine in development. But I have just deployed the app on Vercel, and when it comes to executing this function, firstly only a few followings are console-logged, then the function gets stuck: nothing is logged. Only when I refresh the page (which triggers /api/auth/session serverless function to get called), a few more are logged.

I guess this is because Vercel 10s limits execution for functions. Please advise how I should handle this. Thanks in advance.

background-task serverless next.js
2 Answers

Unfortunately there is not much you can do if you want to keep using nextjs with Vercel. You can host it in aws amplify, but by the time of this answer the lambda functions are not working out of the box with nextjs 12. There are others services out there to help you hosting it in aws, but honestly you should be better of either not using nextjs api routes or, if you really want to use it, create an external app for doing the long process, triggered by the nextjs functions, and request the results to this app or to some database straight from the client side.

This is Vercel's solution for the problem, which in this situation is not very helpfull https://vercel.com/guides/what-can-i-do-about-vercel-serverless-functions-timing-out

sorold0

cron jobs, serverless lambda aws

Snowmanzzz0
Related