Firebase Download Image using Download URL (Without Calling Storage)

Firebase's documentation covers downloading an image if you call storage and getDownloadURL, and I have this working fine (straight from the docs):

storageRef.child('images/stars.jpg').getDownloadURL().then(function(url) {
  // `url` is the download URL for 'images/stars.jpg'

  // This can be downloaded directly:
  var xhr = new XMLHttpRequest();
  xhr.responseType = 'blob';
  xhr.onload = function(event) {
    var blob = xhr.response;
  };
  xhr.open('GET', url);
  xhr.send();

  // Or inserted into an <img> element:
  var img = document.getElementById('myimg');
  img.src = url;
}).catch(function(error) {
  // Handle any errors
});

However, I already have a URL and want to download an image without calling firebase storage. This is my attempt:

var url = "https://firebasestorage.googleapis.com/v0/b/somerandombucketname..."
console.log(url);
// This can be downloaded directly:
var xhr = new XMLHttpRequest();
xhr.responseType = 'blob';
xhr.onload = function(event) {
  var blob = xhr.response;
};
xhr.open('GET', url);
xhr.send();

But, no file is downloaded and no error is shown in the browser's development tools.

Note: I do know the URL is correct because if I put URL directly into my browser search bar, I am able to access the file and download it.

Does anyone know how to download an image using a Download URL that you already have (without calling Firebase Storage as they do in the docs)?