Call hierarchy of async functions inside a loop?

although many answers already, still I believe async/await is the cleanest way.

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function

and you might need babel

https://babeljs.io/

JS async logic syntax changed from callback to promise then to async/await, they all do the same thing, when callback nests a lot we need something like a chain, then promise come, when promise goes in loop, we need something make the chain more plain more simple, then async/await come. But not all browsers support the new syntax, so babel come to compile new syntax to old syntax, then you can always code in new syntax.

getData().then((data) => {
  //do something with final data
})

async function getData() {
  var query_results = [];

  var limit_hit = true;
  var start_from = 0;
  
  //when you use await, handle error with try/catch
  try {
    while (limit_hit) {
      const result = await loadPage(start_from)
      limit_hit = result.limit_hit;
      start_from = result.results.length;
      query_result.push(result.results);
    }
  } catch (e) {
    //when loadPage rejects
    console.log(e)
    return null
  }
  return query_result
}

async function loadPage(start_from) {
  //when you use promise, handle error with reject
  return new Promise((resolve, reject) => Server.Query({
    start_from
  }, (result, err) => {
    //error reject
    if (err) {
      reject(err)
      return
    }
    resolve(result)
  }))
}

If you want to use a loop then I think there is no (clean) way to do it without Promises.

A different approach would be the following:

var query_results = [];

var start_from = 0;

funciton myCallback(result) {  
  if(!result) {
    //first call
    Server.Query({ start_from: start_from}, myCallback);
  } else {
    //repeated call
    start_from = result.results.length
    query_result.push(result.results);
    
    if(!result.limit_hit) {
      //limit has not been hit yet
      //repeat the query with new start value
      Server.Query({ start_from: start_from}, myCallback);
    } else {
        //call some callback function here
    }
  }
}

myCallback(null);

You could call this recursive, but since the Query is asynchronous you shouldn't have problems with call stack limits etc.

Using promises in an ES6 environment you could make use of async/await. Im not sure if this is possible with dojo.

You don't understand callbacks until you have written a rate limiter or queue ;) The trick is to use a counter: Increment the counter before the async request, and decrement it when you get the response, then you will know how many requests are "in flight".

If the server is choked you want to put the item back in the queue.

There are many things you need to take into account: What will happen to the queue if the process is killed ? How long to wait before sending another request ? Make sure the callback is not called many times ! How many times should you retry ? How long to wait before giving up ? Make sure there are no loose ends ! (callback is never called)

When all edge cases are taken into account you will have a rather long and not so elegant solution. But you can abstract it into one function! (that returns a Promise or whatever you fancy). If you have a user interface you also want to show a loading bar and some statistics!

You must await for the server response every time. Here a encapsulated method

var query = (function(){
  var results = [];
  var count = 0;
  return function check(fun){
    Server.Query({ start_from: count}, function(d){
      count = d.results.length;
      results.push(d.results);
      if (d.limit_hit && fun) fun(results);
      else check(fun);
    });
  };
})();

// Call here
var my_query = query(function(d){
  // --> retrive all data when limit_hit is true)
});

You can use a generator function Generators to achieve this For POC:

some basics - You define a generator with an asterick * - it exposes a next function which returns the next value - generators can pause with yield statement internally and can resume externally by calling the next() - While (true) will ensure that the generator is not done until limit has reached

function *limitQueries() {
  let limit_hit = false;
  let start_from  = 0;
  const query_result = [];

  while (true) {
    if (limit_hit) {break;}
    yield Server.Query(params={start_from : start_from}, 
        callback=function* (result) {
        limit_hit = result.limit_hit;
        start_from = result.results.length;
        yield query_result.push(result.results);
    }
  }
}

So apparently, the generator function maintains its own state. Generator function exposes two properties { value, done } and you can call it like this

const gen = limitQueries();
let results = [];
let next = gen.next();

while(next.done) {
  next = gen.next();
}
results = next.value;

You might have to touch your Server.Query method to handle generator callback. Hope this helps! Cheers!