ASP.NET Core File Download No Progress

I have a ASP.NET Core 6 application with a zip dowload function. Here is my Backend Code:

        [HttpGet("/api/DownloadCenterController/DownloadDataset/{fileName}")]
        public IActionResult DownloadDataset(string fileName)
        {
            dynamic response = new ExpandoObject();

            try
            {
                response.status = "200";
                var path = ConfigManager.GetDownloadCenterFinishedZippedDataSetsDirectory() + $"{fileName}";
                var fileInfo = new FileInfo(path);
                this.Response.ContentLength = fileInfo.Length;
                var zipBytes = System.IO.File.ReadAllBytes(path);

                return File(zipBytes, "application/octet-stream", "export.zip");
            }
            catch (Exception ex)
            {
                response.status = "400";
                response.message = "Couldn't download the zip file, error in logs";
                _logger.LogError(ex, "Error download the data set:");
            }

            return Json(response);
        }

And here is the jquery:

// Starts the download of a dataset zip file
async function downloadDataZip(filename) {
    try {
        $.ajax({
            type: "GET",
            url: "/api/DownloadCenterController/DownloadDataset/" + filename,
            //contentType: "application/json",
            contentType: "application/octet-stream",
            xhrFields: {
                responseType: 'blob'
            },
            //data: JSON.stringify(obj),
            success: async function (data) {
                var a = document.createElement('a');
                var url = window.URL.createObjectURL(data);
                a.href = url;
                a.download = 'filename.zip';
                document.body.append(a);
                a.click();
                a.remove();
                window.URL.revokeObjectURL(url);
            }
        });
    } catch (error) {
        console.error(error);
        return undefined;
    }
}

The download is working and the downloaded 200MB zip file is not corrupted, but there is no progress in the browser. So currently, I press the download button - nothing "happens" for like 15 seconds and then the browser opens the download tab, shows a progress of the download for a ms and then boom, the file has already been downloaded. So I assume the file is being downloaded, but the browser does not show any progress but instead only pops up when the download has finished.

What am I doing wrong? I'd like the "normal" download progress in the browser.

Thanks in advance!