What I'm trying to do is: given reddit users, I'd like to iterate over them and fetch their comments and extract the subreddits where the comment was written, and this is really strange:
If I iterate over 20 or more users at the same time, I get a really big error, however, if 19 or less, there's no problem. The rate limit doesn't matter here, I kept logging it through testing, it's not the problem.
This is a very small part of the biiiig error message:
nel: '{"report_to": "w3-reporting-nel", "max_age": 14400, "include_subdomains": false, "success_fraction": 0.2, "failure_fraction": 0.2}'
}
},
body: { message: 'Forbidden', error: 403 },
[Symbol(kCapture)]: false,
[Symbol(kHeaders)]: {
connection: 'close',
'content-length': '38',
'x-ua-compatible': 'IE=edge',
expires: '-1',
'cache-control': 'private, s-maxage=0, max-age=0, must-revalidate, no-store',
'x-ratelimit-remaining': '223.0',
'x-ratelimit-used': '377',
'x-ratelimit-reset': '117',
'content-type': 'application/json; charset=UTF-8',
'access-control-allow-origin': '*',
'access-control-expose-headers': 'X-Moose',
'x-moose': 'majestic',
'accept-ranges': 'bytes',
date: 'Tue, 20 Sep 2022 20:28:04 GMT',
via: '1.1 varnish',
'strict-transport-security': 'max-age=31536000; includeSubdomains',
'x-content-type-options': 'nosniff',
'x-frame-options': 'SAMEORIGIN',
'x-xss-protection': '1; mode=block',
'set-cookie': [
First part of the code:
console.log(`=== ${r.ratelimitRemaining} ===`);
const promises = [];
const dataToWrite: { user: string; subreddits: string[] }[] = [];
// HERE IS THE PROBLEM, IF I SET 25 TO LET'S SAY 19, THEN THERE'S NO ERROR
for (let i = 0; i < 25; i++) {
promises.push(
getUserSubredditsFromComments(users[i]).then((subreddits) => {
console.log(subreddits.length);
})
);
}
void Promise.all(promises).then(() => {
console.log('dataToWrite.length:', dataToWrite.length);
console.log(`=== ${r.ratelimitRemaining} ===`);
});
The function:
export const getUserSubredditsFromComments = (user: RedditUser) => {
const subreddits: string[] = [];
return user
.getComments({ limit: 100 })
.then((comments) => {
for (let j = 0; j < comments.length; j++) {
const subredditName = comments[j].subreddit.display_name;
if (!subreddits.includes(subredditName)) {
subreddits.push(subredditName);
}
}
})
.then(() => subreddits);
};