Throttling API AJAX requests in Node (getting rate limited) - Using AXIOS for AJAX calls...

Viewed 1011

I am getting rate limited making too many requests per second to an API.

I am importing contacts into my database and mapping the data to my DB's schema.

The API initially returns its contact 'groups' (ie categories) as an array of links which need to all have separate API calls made in order to retrieve the actual value.

I must iterate over every contact AND make separate API calls for every link in the array associated with each contact. I quickly hit my rate limit (max 10 per second according to API docs) and get a bunch of incomplete entries and errors as a result.

I am using Axios library to make the requests in Node.

My code is below - any help solving this dilemma would be greatly appreciated - thank you!

        .then(authUser => {

            // RETURN A LIST OF CONTACTS FROM API

            axios.get('url', {
              headers: {
                Authorization: credentials
              }
            })
            .then(response => {

              // MAP OVER EACH CONTACT
              response.data.map( contact => {

                let groups = contact.groupLinks.map(group => {

                    // MAP OVER EACH LINK IN EACH CONTACTS GROUP ARRAY AND MAKE AJAX REQUEST. 
                    return axios.get(`${group.url}/?alt=json`, {
                      headers: {
                        Authorization: 'credentials'
                      }
                    })
                    .then(result => {
                      return result.data
                    })
                    .catch(err => {
                      console.error('FETCH GROUP MEMBERSHIP FAILURE', err.response.data.error)
                      next(err)
                    })
                  })

                return Promise.all(groups)
                .then( response => {
                  return [contact, response]
                })
2 Answers
Related