Snoowrap + Reddit API: Is there a limit on the number of parallel requests or what's this?

Viewed 15

What I'm trying to do is: given reddit users, I'd like to iterate over them and fetch their comments and extract the subreddits where the comment was written, and this is really strange:

If I iterate over 20 or more users at the same time, I get a really big error, however, if 19 or less, there's no problem. The rate limit doesn't matter here, I kept logging it through testing, it's not the problem.

This is a very small part of the biiiig error message:

        nel: '{"report_to": "w3-reporting-nel", "max_age": 14400, "include_subdomains": false, "success_fraction": 0.2, "failure_fraction": 0.2}'
      }
    },
    body: { message: 'Forbidden', error: 403 },
    [Symbol(kCapture)]: false,
    [Symbol(kHeaders)]: {
      connection: 'close',
      'content-length': '38',
      'x-ua-compatible': 'IE=edge',
      expires: '-1',
      'cache-control': 'private, s-maxage=0, max-age=0, must-revalidate, no-store',
      'x-ratelimit-remaining': '223.0',
      'x-ratelimit-used': '377',
      'x-ratelimit-reset': '117',
      'content-type': 'application/json; charset=UTF-8',
      'access-control-allow-origin': '*',
      'access-control-expose-headers': 'X-Moose',
      'x-moose': 'majestic',
      'accept-ranges': 'bytes',
      date: 'Tue, 20 Sep 2022 20:28:04 GMT',
      via: '1.1 varnish',
      'strict-transport-security': 'max-age=31536000; includeSubdomains',
      'x-content-type-options': 'nosniff',
      'x-frame-options': 'SAMEORIGIN',
      'x-xss-protection': '1; mode=block',
      'set-cookie': [

First part of the code:

console.log(`=== ${r.ratelimitRemaining} ===`);

const promises = [];
const dataToWrite: { user: string; subreddits: string[] }[] = [];

// HERE IS THE PROBLEM, IF I SET 25 TO LET'S SAY 19, THEN THERE'S NO ERROR
for (let i = 0; i < 25; i++) {
    promises.push(
        getUserSubredditsFromComments(users[i]).then((subreddits) => {
            console.log(subreddits.length);
        })
    );
}

void Promise.all(promises).then(() => {
    console.log('dataToWrite.length:', dataToWrite.length);
    console.log(`=== ${r.ratelimitRemaining} ===`);
});

The function:

export const getUserSubredditsFromComments = (user: RedditUser) => {
    const subreddits: string[] = [];

    return user
        .getComments({ limit: 100 })
        .then((comments) => {
            for (let j = 0; j < comments.length; j++) {
                const subredditName = comments[j].subreddit.display_name;

                if (!subreddits.includes(subredditName)) {
                    subreddits.push(subredditName);
                }
            }
        })
        .then(() => subreddits);
};
0 Answers
Related