Using S3 as a database to store and query application data

user_13429959 0
Viewed 727

What are the limits of using Amazon S3 (or S3-compatible object storages) as primary database for applications.

This library looks promising with this API:

// Setup Dyno
Dyno dyno = DynoClientBuilder
    .simple()
    .withEndpointConfig("s3.wasabi.sys", "us-east-1")
    .withCredentials(ACCESS_KEY, SECRET_KEY)
    .withBucket("dyno")
    .withKeySpace(":")
    .withBufferSize(1024)
    .build();

// Here's a sample way to create a "User" entity with Dyno

// First create an entity with user_id this will prevent creation of another user 
// with the same username

Entity user = EnityBuilder
    .create(dyno)
    .with("username", "dino")
    .with("user_id")
    .build(uuid(), String.class)
    .putIfAbsent();

And further with this code:

// Since the username "dino" has been secured we can assign the password simply by 
// puting a new entity with the assigned password:

Key key = EnityBuilder
    .create(dyno)
    .with("user_id", user.getValueString())
    .with("password")
    .build(sha256("the_password"), String.class)
    .putIfAbsent();

Again, for the question, what are the limits of using S3? What can make it feasible to use S3 as a database where you can do standard CRUD operations and queries?

  • Create, read, update and delete
  • Query "fields" given conditions (field equals, min-max, contains)
  • Regex search (text search)
amazon-web-services amazon-s3 database java
2 Answers

Amazon S3 is actually a very large NoSQL database. The filename is the Key and the contents is the Value.

However, the contents of an object is immutable, so if you want to 'update' some data, you would need to completely replace the contents of an object.

For example, Amazon.com actually uses S3 as a read-only NoSQL database for historical orders. Once orders are one year old, they cannot be returned/changed. Therefore, the data is exported to S3 and removed from the database. It is slightly slower to query, but old orders are rarely accessed so this is acceptable.

Amazon S3 has a feature called S3 Select that allows SQL to be performed against individual objects. This is useful when seeking a few lines out of a large object, without having to download the object.

Amazon Athena is also great for querying data stored in Amazon S3. It is based on Presto, and allows complex SQL operations against multiple objects and supports columnar formats (Parquet, ORC), partitioning and compression.

user_1747770
user_14355430
Related