Intelligent Tiering Issues

TechNewbie 0
Viewed 18

I was hoping to get other's perspective on something that we've done and beginning to realize it was not the best idea.

Here's some information about our "environment":

  • Account A: We have an AWS account that acts as a data lake (we upload transaction data to S3 daily)
  • Account B: We have another AWS account that our business partners use to access the data in Account A

A few months back, we enabled Intelligent Tiering in S3 where objects are moved to Archive and Deep Archive in 90 and 180 days, respectively. We're now seeing the downfall of this decision. OUr business partners are unable to query data (in account A) from 3 months ago in Athena (account B). Oof.

I guess we did not understand the purpose of intelligent tiering and had hoped that Athena would be able to move tiered objects back into standard s3 when someone queries the data (as in instant retrieval).

There's definitely some use cases that we missed in vetting intelligent tiering.

I am curious how are others leveraging intelligent tiering? Are you only tiering objects that your business partners do not need as "instant retrieval"?

amazon-athena amazon-s3 amazon-web-services
1 Answers

If you goal is to reduce Storage costs, it is worth investigating and understanding the various Storage Classes offered by Amazon S3.

They generally fall into three categories:

  • Instantly available: This is the 'standard' class
  • Instantly available, lower storage cost but higher retrieval cost: This is the 'Infrequent Access' classes. They can be cheaper for data that is only accessed once per month or less. If they are accessed more often, then the Request charges outweigh the savings in Storage costs.
  • Archived: This is typically the Glacier classes. Avoid them if you want to use Amazon Athena.

See that table on: Comparing the Amazon S3 storage classes

For your use-case, you might consider keeping data in Standard by default (since it is heavily accessed), and then move data older than 90 days to S3 One Zone - Infrequent Access. It will still be accessible, but will have a lower storage cost if rarely used.

I would also recommend converting your data to Snappy-compressed Parquet format (preferably partitioned), which will reduce the amount of storage required and will allow Athena to selectively pick which objects it needs to access. It will will also make Athena run faster and reduce the cost of Athena queries.

See: Top 10 Performance Tuning Tips for Amazon Athena | AWS Big Data Blog

John Rotenstein0
Related