How to partition high cardinality data that is extended daily?

Viewed 86

I have an Athena table like this:

values_by_time:

  • Columns:
    • id string (contains UUIDs)
    • value string
    • created timestamp (this is a partition key)
  • Partition: [created]

But I want to query like this:

SELECT * FROM values_by_time WHERE id = 'ac09b809-471e-4296-b99b-e57408075609'

So my partition by [created] is inefficient.

I read that bucketing can be used for high cardinality columns, such as my id:

values_by_id:

  • Columns:
    • id string (bucket by this)
    • created timestamp
    • value string
  • Bucket By: [id]

This makes the look-up efficient:

SELECT * FROM values_by_id WHERE id = 'ac09b809-471e-4296-b99b-e57408075609'

However, I cannot do an INSERT INTO on a table with bucketing! This means that I cannot add new data to it after table creation. Creating the table fresh every day seems inefficient too.

How should I arrange my Athena table so that:

  1. I can efficiently query by id
  2. I can insert new data every day
0 Answers
Related