Stop request after X-amount is fetched

Question

I want to fetch an external, third-party api from client side. But the feed is huge (160,000 objects in one array) I have no control over the external api.

Is it possible to set a max-limit, to amount of bytes maybe? So I could fetch about 50,000 objects in this case? Cause it is loading way too long

I have been reading about axios and the maxContentLength and maxBodyLength but without any luck - it doesn't have any effect.

Also, the api doesn't support any pagination like ?size=20&page=1

So it is possible using XMLHttpRequest, I could tell in the comments. But is the same solution possible using fetch function? Since we don't have this e.loaded

user_1229023 · Answer

Ok, here's a small-and-dirty PoC just to test the idea (kudos to Anime News Network for that wonderful and truly public API):

async function limitedFetch(url, maxBytes) {
  return new Promise((resolve, reject) => {
    try {
      const xhr = new XMLHttpRequest();
      xhr.onprogress = (ev) => {
        console.log(`Received ${ev.loaded} bytes`);
        if (ev.loaded < maxBytes) return;
        resolve(ev.target.responseText);
        xhr.abort();
      };
      xhr.onload = (ev) => {
        resolve(ev.target.responseText);
      };
      xhr.onerror = (ev) => { reject(new Error(ev.target.status)) }

      xhr.open('GET', url);
      xhr.send();
    }
    catch (err) {
      reject(err);
    }
  });
}

const out = document.querySelector('pre');
const filter = document.querySelector('#filter');
document.querySelectorAll('button').forEach(
  btn => btn.addEventListener('click', async (ev) => {
    const rateLimit = ev.target.dataset.rate || Infinity;
    const cacheBuster = '&' + Date.now();
    const animeApiUrl = 
    `https://cdn.animenewsnetwork.com/encyclopedia/api.xml?title=~${filter.value}`;
    out.textContent = 'Waiting...';
    console.clear();
    try {
      const response = await limitedFetch(animeApiUrl + cacheBuster, rateLimit);
      out.textContent = `Received ${response.length} characters:
  ${response}`;
    }
    catch (err) {
      out.textContent = `Oh noes! ${err.message}
  ${err.stack}`;
    }
  })
);

The key here is using onprogress event of XMLHttpRequest to track the count of bytes received (that's loaded property on ProgressEvent object). When it overflows the limit, the fetcher resolves immediately - and drops the corresponding request.

Play with different release buttons, and you'll see the difference in both bytes received overall - and the number of console.log messages (those are fired for each received chunk of data).

Now the catch: with default filter (just 'a') 'rated' requests never stop at exactly 1000 bytes; that's not how chunks work. Instead there's always one chunk not greater than ~50K bytes; with 'rated' request, that's the only one, with unrated, there'll be three more or so.

Stop request after X-amount is fetched

1 Answers