createBlockBlobFromStream() is overriding existing content instead of appending

user_12354698 0
Viewed 172

We work with very large files so we devide them into chunk of 20mb then call the upload function to upload them to blob storage.I am calling upload() in node js where I found that I am missing something while upload. Only 20mb is getting uploaded each time , I doubt nodejs is overriding the content rather than appending the stream.

can somebody help to fix it?

const chunkSize = Number(request.headers["x-content-length"]);
const userrole = request.headers["x-userrole"];
const pathname = request.headers["x-pathname"];

var form = new multiparty.Form();
form.parse(request, function (err, fields, files) {
if (files && files["payload"] && files["payload"].length > 0) {
var fileContent = fs.readFileSync(files["payload"][0].path);
// log.error('fields',fields['Content-Type'])
fs.unlink(files["payload"][0].path, function (err) {
if (err) {
log.error("Error in unlink payload:" + err);
}
});
var size = fileContent.length;
if (size !== chunkSize) {
sendBadRequest(response, "Chunk uploading was not completed");
return;
}
//converting chunk[buffers] to readable stream
const stream = Readable.from(fileContent);
var options = {
contentSettings: {
contentType: fields['Content-Type']
}
}
blobService.createBlockBlobFromStream(containerName, pathname, stream, size, options, error => {
});

Headers:

X-id: 6023f6f53601233c080b1369
X-Chunk-Id: 38
X-Content-Id: 43bfdbf4ddd1d7b5cd787dc212be8691d8dd147017a2344cb0851978b2d983c075c26c6082fd27a5147742b030856b0d71549e4d208d1c3c9d94f957e7ed1c92
X-pathname: 6023f6ae3601233c080b1365/spe10_lgr311_2021-02-10_09-08-37/output/800mb.inc

Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryqodrlQNFytNS9wAc
X-Content-Name: 800mb.inc
azure-blob-storage angular express node.js
1 Answers

The issue is that you're using createBlockBlobFromStream which will overwrite the contents of a blob. This method is used to create a blob in a single request (i.e. complete blob data is passed as input). From the documentation here:

Uploads a block blob from a stream. If the blob already exists on the service, it will be overwritten. To avoid overwriting and instead throw an error if the blob exists, please pass in an accessConditions parameter in the options object.

In your case, you're uploading chunks of the data. What you would need to do is use createBlockFromStream method for each chunk that you're uploading.

Once all chunks are uploaded, you would need to call commitBlocks method to create the blob.

UPDATE

How can we generate the blockId?

A block id is simply a string. Key thing to remember is that when you're calling createBlockFromStream, the length of block id of each block you're sending must be the same. You can't use 1,2,3,4,5,6,7,8,9,10,11... for example. You will have to use something like 01,02,03,04,05,06,07,08,09,10,11... so that they're of same length. You can use GUID for that purpose. Also, the maximum length of the block id is 50 characters.

should it be unique for all?

Yes. For a blob, the block ids must be unique otherwise it will overwrite the content of a previous block uploaded with the same id.

can u show the example code so that i can try similar way to implement?

Please take a look here.

Basically the idea is very simple: On your client side, you're chunking the file and sending each chunk separately. What you will do is apart from sending that data, you will also send a block id string. You will also need to keep that block id on your client side. You will repeat the same process for all the chunks of your file.

Once all chunks are uploaded successfully, you will make one more request to your server and send the list of all the block ids. Your server at that time will call commitBlocks to create the blob.

Please note that the order of block ids in your last request is important. Azure Storage Service will use this information to stitch the blob together.

user_1880960
Related