Cannot delete item from a versioned s3 bucket

Question

I am trying to delete a specific .jpg file from an s3 bucket. I have tried numerous different methods such as delete_object and several other stack overflow answers. None of them have worked properly and result in the program crashing with differing exit codes each time. The code below doesn't crash the program but the bucket remains unchanged. Does anyone know how to fix this?

s3 = boto3.resource('s3')
s3.Object('my-bucket', 'label_file').delete()

When printing the response of the call, this is the message printed to the terminal.


{
  'ResponseMetadata': {
    'RequestId': 'FJEOSIJFIO2J31',
    'HostId': 'VEt+tpQ+DSAW5UTx/+kGgmE0JX43345lX2ULHMDSDrKye1PsTA3Eo=',
    'HTTPStatusCode': 314,
    'HTTPHeaders': {
      'x-amz-id-2': 'VEt+tpQo2DAS6z+m8o3215UTx/+kGgmE0JX2DSi3ULHMDrKyevPsTA3Eo=',
      'x-amz-request-id': 'T255WdZ732sVDWJF',
      'date': 'Wed, 07 Apr 2021 19:28:04 GMT',
      'x-amz-version-id': '4gi8EtDS27ZNy221tkziO',
      'x-amz-delete-marker': 'true',
      'server': 'AmazonS3'
    },
    'RetryAttempts': 0
  },
  'DeleteMarker': True,
  'VersionId': '4gioU7FD36t5hcX21IOtkziO'
}

user_6106791 · Answer

Your S3 bucket has versioning enabled, because you have:

      'x-amz-delete-marker': 'true',
   ...
   'DeleteMarker': True,

in the response.

A delete marker in Amazon S3 is a placeholder (or marker) for a versioned object that was named in a simple DELETE request. Because the object is in a versioning-enabled bucket, the object is not deleted. But the delete marker makes Amazon S3 behave as if it is deleted.

So, basically, you have to delete the marker, and then delete the file.

If you want to delete a delete marker, it must have a version ID, and you must specify that ID in a DELETE Object versionId request. If you use a DELETE request to delete a delete marker (without specifying the version ID of the delete marker), Amazon S3 does not delete the delete marker, but instead, inserts another delete marker.

More on removing the delete marker

To remove the delete markers and the versioned files you can use this code:

WARNING! This will effectively purge your bucket!

def remove_versions(target_bucket: str) -> None:
    s3 = boto3.resource("s3", region_name="your_s3_bucket_region_goes_here")
    bucket = s3.Bucket(target_bucket)
    object_version_iterator = bucket.object_versions.all()

    for object_version in object_version_iterator:
        key = object_version.object_key
        object_id = object_version.id
        print(f"Deleting key: {key} with id: {object_id}")
        object_version.delete()

    print(f"Done. {target_bucket} has been purged.")

user_2246345 · Answer

You can delete individual object versions as well.

object_versions

import boto3
bucket_name = "mybucket"
objects_to_delete = ['label_file', 'label_file1']

bucket = boto3.resource('s3').Bucket(bucket_name)
versions = bucket.object_versions.all()
# if you wanna delete all of them removing the check will be enough
for version in versions:
    if version.object_key in objects_to_delete:
        version.delete()

If you want you can create a filter for deletion as well using filter

object_summary_iterator = bucket.objects.filter(
    Delimiter='string',
    EncodingType='url',
    Marker='string',
    MaxKeys=123,
    Prefix='string',
    RequestPayer='requester',
    ExpectedBucketOwner='string'
)

There is another good example here Delete all versions of an object in S3 using python?

Cannot delete item from a versioned s3 bucket

2 Answers