Error uploading 200MB file to Azure blob storage

user_10355901 0
Viewed 536

I am using Azure.Storage.Blobs v12.6.0.0 with a small project that upload a file once a day to azure blob storage, but I have not been able to upload large files.

Actually I am testing with 249MB file using the below code and it fails. If the file is small it get uploaded with no issue but with the large file after it starts to upload it fails after about 10 minutes with the error Retry failed after 6 tries.

My internet speed is 40Mbps

Here is the code I use (it is VB.net in Visual Studio 2019):

Using fStream As FileStream = File.OpenRead(fPath)
        Dim credential As New StorageSharedKeyCredential("account", storageAccountKey)
        Dim serviceURI As New Uri("https://name.blob.core.windows.net/")
        Dim blobSC As New BlobServiceClient(serviceURI, credential)

        Dim blobContainer As BlobContainerClient = blobSC.GetBlobContainerClient("files")

        Dim blob As BlobClient = blobContainer.GetBlobClient(fName)

        Dim t As Task = blob.UploadAsync(fStream, True, Threading.CancellationToken.None)

        t.Wait()
End Using

Thanks in advance for any help.

azure-blob-storage azure-storage azure visual-studio
1 Answers

I already solved my problem, the issue was I need to split the upload into blocks and also there were some uncommitted blocks in my account with same blob name.

I had to upload an small blob with same name to make azure storage to clear the uncommitted blob blocks.

Here is the code I use now:

Using fStream As FileStream = File.OpenRead(fPath)
    Dim bytesFile As Byte()

    Using ms As MemoryStream = New MemoryStream
        fStream.CopyTo(ms)
        bytesFile = ms.ToArray()
    End Using

    Dim containerClient As BlobContainerClient = New BlobContainerClient(storageConString, "ContainerName")
    Dim blobClient As BlockBlobClient = containerClient.GetBlockBlobClient("BlobName")

    Dim blockId As Integer = 1000

    Do
        Dim bytesToCopy As Long = Math.Min(bytesRemain, pageSizeInBytes)
        Dim bytesToSend As Byte() = New Byte(bytesToCopy - 1) {}


        Array.Copy(bytesFile, prevLastByte, bytesToSend, 0, bytesToCopy)
        prevLastByte += bytesToCopy
        bytesRemain -= bytesToCopy

        Dim base64BlockId As String = Convert.ToBase64String(Encoding.UTF8.GetBytes(blockId))
        Dim stageResponse = blobClient.StageBlock(base64BlockId, New MemoryStream(bytesToSend))
        Dim responseInfo = stageResponse.GetRawResponse()

        blocklist.Add(base64BlockId)

        EventLog1.WriteEntry("Respoonse for blockId: " + blockId.ToString + vbCrLf + vbCrLf + responseInfo.ToString, EventLogEntryType.Warning)

        blockId += 1
    Loop While (bytesRemain > 0)

    Dim contentResponse = blobClient.CommitBlockList(blocklist)
End Using
user_103559010
Related