How to upload a 10 Gb file using SAS token

user_1269281 0
Viewed 663

I'm trying to upload a large file (over 10Gb) to Azure Blob Storage using SAS tokens. I generate the tokens like this

    val storageConnectionString = s"DefaultEndpointsProtocol=https;AccountName=${accountName};AccountKey=${accountKey}"
    val storageAccount = CloudStorageAccount.parse(storageConnectionString)
    val client = storageAccount.createCloudBlobClient()

    val container = client.getContainerReference(CONTAINER_NAME)
    val blockBlob = container.getBlockBlobReference(path)
    val policy = new SharedAccessAccountPolicy()
    policy.setPermissionsFromString("racwdlup")
    val date = new Date().getTime();
    val expiryDate = new Date(date + 8640000).getTime()
    policy.setSharedAccessStartTime(new Date(date))
    policy.setSharedAccessExpiryTime(new Date(expiryDate))
    policy.setResourceTypeFromString("sco")
    policy.setServiceFromString("bfqt")
    val token = storageAccount.generateSharedAccessSignature(policy)

Then I tried the Put Blob API and hit the following error

$ curl -X PUT -H 'Content-Type: multipart/form-data' -H 'x-ms-date: 2020-09-04' -H 'x-ms-blob-type: BlockBlob' -F file=@10gb.csv  https://ACCOUNT.blob.core.windows.net/CONTAINER/10gb.csv\?ss\=bfqt\&sig\=.... -v

< HTTP/1.1 413 The request body is too large and exceeds the maximum permissible limit.
< Content-Length: 290
< Content-Type: application/xml
< Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
< x-ms-request-id: f08a1473-301e-006a-4423-837a27000000
< x-ms-version: 2019-02-02
< x-ms-error-code: RequestBodyTooLarge
< Date: Sat, 05 Sep 2020 01:24:35 GMT
* HTTP error before end of send, stop sending
< 
<?xml version="1.0" encoding="utf-8"?><Error><Code>RequestBodyTooLarge</Code><Message>The request body is too large and exceeds the maximum permissible limit.
RequestId:f08a1473-301e-006a-4423-837a27000000
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
Time:2020-09-05T01:24:35.7712576Z</Message><MaxLimit>268435456</MaxLimit></Error>%

After that tried uploading it using PageBlob (I saw in the documentation something like size can be up to 8 TiB)

$ curl -X PUT -H 'Content-Type: multipart/form-data' -H 'x-ms-date: 2020-09-04' -H 'x-ms-blob-type: PageBlob' -H 'x-ms-blob-content-length: 1099511627776' -F file=@10gb.csv  https://ACCOUNT.blob.core.windows.net/CONTAINER/10gb.csv\?ss\=bfqt\&sig\=... -v

< HTTP/1.1 400 The value for one of the HTTP headers is not in the correct format.
< Content-Length: 331
< Content-Type: application/xml
< Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
< x-ms-request-id: b00d5c32-101e-0052-3125-83dee7000000
< x-ms-version: 2019-02-02
< x-ms-error-code: InvalidHeaderValue
< Date: Sat, 05 Sep 2020 01:42:24 GMT
* HTTP error before end of send, stop sending
< 
<?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidHeaderValue</Code><Message>The value for one of the HTTP headers is not in the correct format.
RequestId:b00d5c32-101e-0052-3125-83dee7000000
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
Time:2020-09-05T01:42:24.5137237Z</Message><HeaderName>Content-Length</HeaderName><HeaderValue>10114368132</HeaderValue></Error>%

Not sure what is the proper way to go about uploading such large file?

azure-blob-storage azure
2 Answers

Check the different blob types here: https://docs.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs

The page blob actually limits the maximum size to 8TB but it's optimal for for random read and write operation.

On the other hand:

Block blobs are optimized for uploading large amounts of data efficiently. Block blobs are comprised of blocks, each of which is identified by a block ID. A block blob can include up to 50,000 blocks.

So block blobs is the way to go as it supports sizes of up to 190.7 TB (preview mode)

Now you need to use the put block https://docs.microsoft.com/en-us/rest/api/storageservices/put-block to upload the blocks that will form your blob.

user_16436760
user_866110
Related