Unable to delete Multiple Blobs from Container in Python

user_12171714 0
Viewed 1373

I am trying to delete blobs in my containers. Each container has minimum 1500-2000 blobs. Each container contains jpeg files and one mp4 file. If the mp4 file is present, only then I will delete the blobs inside that particular container. Every time when I try to execute content.delete_blobs(*blobsToDelete) ,I get the following exception:

Exception in Non AI : Traceback (most recent call last):
  File "deleteblobs.py", line 278, in BlobsToDeleteeNonAI
    content.delete_blobs(*blobsToDelete)
  File "C:\Users\hp\AppData\Local\Programs\Python\Python38-32\lib\site-packages\azure\core\tracing\decorator.py", line 83, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "C:\Users\hp\AppData\Local\Programs\Python\Python38-32\lib\site-packages\azure\storage\blob\_container_client.py", line 1194, in delete_blobs
    return self._batch_send(*reqs, **options)
  File "C:\Users\hp\AppData\Local\Programs\Python\Python38-32\lib\site-packages\azure\storage\blob\_shared\base_client.py", line 304, in _batch_send
    raise error
azure.storage.blob._shared.response_handlers.PartialBatchErrorException: There is a partial failure in the batch operation.

Here is what my code looks like :

def BlobsToDeleteeNonAI():
        blobsToDelete = []
        #Will delete all the photos except the video.
        try:
                for containerName in NonAICandidates:
                        try:
                                mp4Found = 0
                                content = blob_service_client.get_container_client(str(containerName))
                                for blobs in content.list_blobs():
                                        print("\n"+blobs.name)
                                        #file.write("\n" +blobs.name)
                                        if(blobs.name.endswith(".jpeg")):   #str(blobs.size)
                                                blobsToDelete.append(blobs.name)

                                        if(blobs.name.endswith(".mp4")):
                                                mp4Found = 1
                                                file.write("\nMP4 File Name : " +str(blobs.name))
                                #Will only Delete if and only if the Video File is Present
                                if(mp4Found == 1):
                                        #DeleteCodeHere
                                        
                                        file.write("\n Mp4 Found : " +str(mp4Found) + " for " +str(containerName))
                                        #file.write("\n Blobs to Delete : "+str(blobsToDelete))
                                        
                                        content.delete_blobs(*blobsToDelete)
                                        blobsToDelete.clear()
                                        file.write("\n Blobs Deleted for : " +str(containerName))
                                else:
                                        file.write("\nMp4 File Not found for Non AI Candidate : " +str(containerName) + ". Cannot Perform Deletion Operation.");
                                                
                                           
                        except:
                                file.write("\nException in Non AI : " +str(traceback.format_exc()))
                                blobsToDelete.clear()
        except:
                 file.write("\nException : " +str(traceback.format_exc()))




if __name__ == "__main__":

        NonAICandidates = ['container1', 'container2', 'container3', 'container4', 'container5', 'container6', ....]


        BlobsToDeleteeNonAI()

Is there anything wrong with the implementation or is there any other issue which is preventing me from deleting the blobs ?

azure-blob-storage azure-storage azure containers python
3 Answers

The reason is that a single batch can only support 256 sub-requests. And in your container, there are at least 1500-2000 blobs to be deleted, when you try to delete these blob in one delete_blobs method, it's beyond the 256 limitations.

You should modify your code, in one delete_blobs method, only delete 1 to 256 blobs. Here is an sample:

#Will only Delete if and only if the Video File is Present
if(mp4Found == 1):
    blobs_lenth=len(blobsToDelete)

    start=0
    end=256

    while end<=blobs_lenth:
         #each time, delete 256 blobs at most
         container_client.delete_blobs(*blobsToDelete[start:end])
         start = start + 256
         end = end + 256

         if start < blobs_lenth and end > blobs_lenth:
            container_client.delete_blobs(*blobsToDelete[start:blobs_lenth])
user_101858160 
if(mp4Found == 1):
    blobs_lenth=len(blobsToDelete)

    if blobs_lenth <= 256:
        container_client.delete_blobs(*blobsToDelete) 

    else:
        start=0
        end=256

        while end<=blobs_lenth:
             #each time, delete 256 blobs at most
             container_client.delete_blobs(*blobsToDelete[start:end])
             start = start + 256
             end = end + 256

             if start < blobs_lenth and end > blobs_lenth:
                container_client.delete_blobs(*blobsToDelete[start:blobs_lenth])
user_162438640

I found this sample can be simply written like as below. works fine.

if(mp4Found == 1):
    blobs_length=len(blobsToDelete)

    for i in range(0, blobs_length, 256):
        container_client.delete_blobs(*blobsToDelete[i: i+256])
user_149347690
Related