JavaScript: Await for a recursive tree to finish, where each recursive level is an API call

Question

I'm trying to build a JSON tree using recursive API calls but I am having issues with flow control of the data structure. How can I block the flow until the BuildTree function stack ends?

Here is the partial code. Thanks in advance.

//FIRST CALL TO RETRIEVE THE ROOT
function callFW() {
    d3.json(url, function(data) { //?<----- api call
            Tree["uid"] = data.request.uid
            Tree["hid"] = data.firmware.meta_data.hid
            Tree["size"] = data.firmware.meta_data.size
            Tree["children"] = [];
            
            BuildTree(data.firmware.meta_data.included_files,Tree["children"])
            //WAIT FOR BUILDTREE
            console.log(Tree)
        } 
}

The BuildTree function is something like this:

async function BuildTree(included_files, fatherNode){ 
        if( included_files.length > 0){
            promises = [];

            included_files.forEach(function(item) {
                url = endpoint+"file_object/"+item+"?summary=true";
                promises.push(axios.get(url));
            });

            Promise.all(promises).then(function (results) {
                results.forEach(function (response) {
                    
                    var node = {}
                    node["uid"]= response.data.request.uid
                    node["hid"]= response.data.file_object.meta_data.hid
                    node["size"] = response.data.file_object.meta_data.size
                    node["children"] = []

                    fatherNode.push(node)

                    BuildTree(response.data.file_object.meta_data.included_files, node["children"])
                    


                });
            });
        }
    }

user_1021703 · Answer

You are using the async keyword for no reason, because you are not using await inside your function. You might as well use it :)

async function BuildTree(included_files, fatherNode) {
    if (included_files.length > 0) {

        let promises = included_files.map( item => {
            let url = endpoint + "file_object/" + item + "?summary=true";
            return axios.get(url)
        });

        const results = await Promise.all(promises);

        for(let response of results){

            var node = {}
            node["uid"] = response.data.request.uid
            node["hid"] = response.data.file_object.meta_data.hid
            node["size"] = response.data.file_object.meta_data.size
            node["children"] = []

            fatherNode.push(node)

            await BuildTree(response.data.file_object.meta_data.included_files, node["children"]);
        };
    }
};

And then you can simply await BuildTree :

function callFW() {
    d3.json(url, async function(data) {
            Tree["uid"] = data.request.uid
            Tree["hid"] = data.firmware.meta_data.hid
            Tree["size"] = data.firmware.meta_data.size
            Tree["children"] = [];
            
            await BuildTree(data.firmware.meta_data.included_files,Tree["children"]);

            console.log(Tree)
        })
}

user_1243641 · Answer

I think it would be a lot cleaner to make the functions more self-contained. For instance, the main function could replace

    Tree["children"] = [];
        
    await BuildTree(data.firmware.meta_data.included_files,Tree["children"]);

with

    Tree["children"] = await BuildTree (data.firmware.meta_data.included_files)

if we restructure BuildTree a bit. To that end, we can write BuildTree to recursively build the children node of the tree like this:

const endpoint = 'http://my.service/'

const BuildTree = async (files) => Promise .all (
  files .map ((name) => 
    axios .get (`${endpoint}file_object/${name}?summary=true`) 
      .then (async response => ({
        uid: response.data.request.uid,
        hid: response.data.file_object.meta_data.hid,
        size: response.data.file_object.meta_data.size,
        children: await BuildTree (response.data.file_object.meta_data.included_files)
      }))
  )
)


BuildTree (['ca26bcfc'])
  .then  (console .log)
  .catch (console .warn)

.as-console-wrapper {max-height: 100% !important; top: 0}

(Note that there's an uninteresting dummy version of axios involved, faking out service calls, randomly giving hid and size parameters and randomly returning between zero and four children. So every run will give you back different random data, sometimes one level deep and sometimes five or ten or even more levels, but you might have to run a few times to see the nesting.)

The important thing here is how the recursive call is managed. It simply returns a promise for an array of items, so we can slot it in to generate the children just as we do in the call from the main function. This leads to cleaner code.

There is one more thing that I would do to improve this function. It is using the global variable endpoint. That makes it impure, harder to test, and more fragile. Instead, we could pass it in. I would prefer this version:

const BuildTree = async (endpoint, files) => Promise .all (
  files .map ((name) => 
    axios .get (`${endpoint}file_object/${name}?summary=true`) 
      .then (async response => ({
        uid: response.data.request.uid,
        hid: response.data.file_object.meta_data.hid,
        size: response.data.file_object.meta_data.size,
        children: await BuildTree (endpoint, response.data.file_object.meta_data.included_files)
      }))
  )
)

BuildTree ('http://my.service/', ['ca26bcfc'])
  .then (console .log)
  .catch (err => console .log (`Error: ${err}`))

or even:

const BuildTree = (endpoint) => async (files) => Promise .all (
  // ...
        children: await BuildTree (endpoint) (response.data.file_object.meta_data.included_files)
  // ...
BuildTree ('http://my.service/') (['ca26bcfc']) 
  // ...

I would also recommend a similar clean-up of the main function, which takes no arguments, but uses the global variable Tree, again making it much harder to test. But I don't know enough about your environment to offer practical suggestions.

JavaScript: Await for a recursive tree to finish, where each recursive level is an API call

2 Answers