Bulk Downloads in Azure Blob Storage

user_10430240 0
Viewed 1838

I need to figure out a way to let my users download several pdf files (sometimes thousands), from Azure Blob Storage, I know that I can download the files in paralel, and that would make things quicker, but the issue here is that the user could possibly have thousands of pdf files to download, and, that isn't at all reasonable.

Also, I can't download the files to another server, zip them, and let the user download them from there, as that would be incredibly inefficient for me.

Is there a way to create a zip of the files and let the user download that (other than the way above)? I saw other questions on this topic but none gave an answer/solution that suits my needs.

What would be, the absolute best way I can do this? Or isn't there another way to preform this task?

Thank you in advance.

azure-blob-storage azure blob download c#
1 Answers

Since no one gave an answer, and I see more posts about this on stack overflow and other sites, I decided to share here my solution (can't share code, because reasons...)

Firstly, as of today 04-09-2020 there's still no support for bulk download from Azure Blob Storage in a zip (or other format) that is directly from azure to client, without routing the download flow through a server that does the organizing and zipping.

The problem I had...
A need to download (several) files from Azure Blob Storage, zip them (maybe organize them by folders), and prompt the client to download them in bulk without any download data passing through the server and not filling the client downloads folder with scattered files...

During my research I thought about doing everything on the client's side in javascript through memory and let the client download it, but it could be quite memory expensive since my downloads could be in the GB size range.

The solution...
Then I came across a javascript library called StreamSaver, this library writes the files with streams and writes directly on the client's machine, meaning the memory expense was much less.
By luck this library also allows to organize the files inside the 'download directory' that will be prompted to the user, and even lets me zip that directory before telling the user if he wants to download it, meaning that this one library solved, almost, all my problems.

Now I only have a webmethod called by javascript that returns all the Azure SAS url to download from, and the rest is all in javascript in the client.

TL;DR:
Used StreamSaver javascript library to download, organize and zip all the files from the client side and then prompt them to download it, only using a webmethod to get all the urls wich are to be downloaded.

This solution works (from what I've tested) in at least these browsers:

  • Chrome;
  • FireFox;
  • Opera;
  • Edge (Chromium)

Problems I came across using the StreamSaver Library...
There are a few drawbacks/problems with the library,

  • 1st Safary doesn't support it! more info about this here
  • 2nd StreamSaver only allows zipping to files smaller than 4GB, this could be worked around using yet another library for zipping...
user_104302400
Related