S3 eventual consistency for read after write

Question

I read a lot about different scenarios and questions that are about s3 eventual consistency and how to handle it to not get 404 error. But here I have a little bit strange use case/requirement! What I'm doing is writing bunch of files to a temp/transient folder in a s3 bucket (using a spark job and make sure job is not going to fail), then remove the main/destination folder if the previous step is successful, and finally copy files over from temp to main folder in the same bucket. Here is part of my code:

# first writing objects into the tempPrefix here using pyspark
...
# delete the main folder (old data) here
...

# copy files from the temp to the main folder
for obj in bucket.objects.filter(Prefix=tempPrefix):

    # this function make sure the specific key is available for read 
    # by calling HeadObject with retries - throwing exception otherwise
    waitForObjectToBeAvaiableForRead(bucketName, obj.key)

    copy_source = {
        "Bucket": bucketName,
        "Key": obj.key
    }
    new_key = obj.key.replace(tempPrefix, mainPrefix, 1)
    new_obj = bucket.Object(new_key)
    new_obj.copy(copy_source)

This seems to work to avoid any 404 (NoSuchKey) error for immediate read after write. My question is will the bucket.objects.filter give me the newly written objects/keys always? Can eventual consistency affect that as well? The reason I'm asking this because the HeadObject call (in the waitForObjectToBeAvaiableForRead function) sometimes returns 404 error for a key which is returned by bucket.objects.filter!!! I mean the bucket.objects returns a key which is not available for read!!!

user_212514 · Answer

When you delete an object in S3, AWS writes a "delete marker" for the object (this assumes that the bucket is versioned). The object appears to be deleted, but that is a sort of illusion created by AWS.

So, if you are writing objects over previously-existing-but-now-deleted objects then you are actually updating an object which results in "eventual consistency" rather than "strong consistency."

Some helpful comments from AWS docs:

A delete marker is a placeholder (marker) for a versioned object that was named in a simple DELETE request. Because the object was in a versioning-enabled bucket, the object was not deleted. The delete marker, however, makes Amazon S3 behave as if it had been deleted.

If you try to get an object and its current version is a delete marker, Amazon S3 responds with:

A 404 (Object not found) error

A response header, x-amz-delete-marker: true

Specific Answers

My question is will the bucket.objects.filter give me the newly written objects/keys always?

Yes, newly written objects/keys will be included if you have fewer than 1,000 objects in the bucket. The API returns up to 1,000 objects.

Can eventual consistency affect that as well?

Eventual consistency affects the availability of the latest version of an object, not the presence of the object in filter results. The 404 errors are the result of trying to read newly written objects that were last deleted (and full consistency has not yet been achieved).

S3 eventual consistency for read after write

1 Answers