Recursively sending http requests using data from json response

Question

I'm trying to extract all names from an API that returns a JSON with the following structure:

{
    "data" : [
        {
            "attributes" : {
                "name" : "Prog1"
                ...
            },
            "id" : "12345",
        },
        {
            "attributes" : {
                "name" : "Prog2"
                ...
            },
            "id" : "67890",
        },
        ...
    ],
    "links" : {
        "next" : "https://api.company.com/v1/programs?page%5Bnumber%5D=3",
        "self" : "https://api.company.com/v1/programs?page%5Bnumber%5D=2",
        "prev" : "https://api.company.com/v1/programs?page%5Bnumber%5D=1",
    }
}

I want to keep sending requests, and logging the names as long as there is a "next" link, after which I'll stop sending requests.

Here's the function I have so far:

async function getPrograms(url: string, user: string, pass: string) {
    try {
        const response = await fetch(url, { 
            method: 'GET',
            headers: {Authorization: 'Basic ' + Buffer.from(`${user}:${pass}`).toString('base64')},
        });

        if (!response.ok) {
            throw new Error(`Error! status: ${response.status}`);
        }

        console.log(response);

        const result = (await response.json()) as GetProgramsResponse;

        result.data.forEach((program) => {
            // Later will add other stuff here
            console.log(program.attributes.name);
        })

        console.log(result.links.next);

        if (result.links.next) {
            getPrograms(result.links.next, user, pass)
        }

    } catch (error) {
        if (error instanceof Error) {
            console.log('error message: ', error.message);
            return error.message;
        } else {
            console.log('unexpected error: ', error);
            return 'An unexpected error occured';
        }
    }
};

getPrograms(url, u, p);

The program hangs after around 6 iterations, not finishing but not continuing either. I'm assuming it has to do with the requests being sent asynchronously, but I'm relatively new to TypeScript and can't seem to figure out how to wait for all iterations to finish.

Would love for someone to point me to the right direction, and get some feedback regarding the proposed solution as a whole.

Thanks!

kpostekk · Answer

TL;DR You have missing await (floating promise) in this part of code:

if (result.links.next) {
    await getPrograms(result.links.next, user, pass) // I have awaited this call.
}

If you want to know how async/await is handled in typescript/javascript check this article

Also I have found some more issues with this snippet.

Using recursion to call multiple enpoints isn't the most effiecient way to do that. It only increases call stack size and code complexity. In the worst case scenario, program may won't exit from recursion and it will crash because it's call stack is full. I Would suggest using while loop (iterate while link.next exists).
Instead of using someArray.forEach I suggest using classic for const of loop. It doesn't have any particular performance benefits, but it's easier to read. Also you can call async funtion wthin this loop.

for (const program of result.data) {
  await saveToDb(program)
}

Try to narrow try/catch clousre. Having big chunks of code in a singe try/catch is a bad practice.

Recursively sending http requests using data from json response

1 Answers