How do you synchronize related collections in Cosmos Db?

user_53328 0
Viewed 283

My application need to support lookups for invoices by invoice id and by the customer. For that reason I created two collections in which I store the (exact) same invoice documents:

  • InvoicesById, with partition key /InvoiceId
  • InvoicesByCustomerId, with partition key /CustomerId

Apparently you should use partition keys when doing queries and since there are two queries I need two collections. I guess there may be more in the future.

Updates are primarily done to the InvoicesById collection, but then I need to replicate the change to InvoicesByCustomer (and others) as well.

Are there any best practice or sane approaches how to keep collections in sync?

I'm thinking change feeds and what not. I want avoid writing this sync code and risk inconsistencies due to missing transactions between collections (etc). Or maybe I'm missing something crucial here.

azure-cosmosdb
1 Answers

Change feed will do the trick though I would suggest to take a step back before brute-forcing the problem.

Please find detailed article describing split issue here: Azure Cosmos DB. Partitioning.

Based on the Microsoft recommendation for maintainable data growth you should select partition key with highest cardinality (in your case I assume it will be InvoiceId). For the main reason:

Spread request unit (RU) consumption and data storage evenly across all logical partitions. This ensures even RU consumption and storage distribution across your physical partitions.

You don't need creating separate container with CustomerId partition key as it won't give you desired, and most importantly, maintainable performance in future and might result in physical partition data skew when too many Invoices linked to the same customer.

To get optimal and scalable query performance you most probably need InvoiceId as partition key and indexing policy by CustomerId (and others in future).

There will be a slight RU overhead (definitely not multiplication of RUs but rather couple additional RUs per request) in consumption when data you're querying is distributed between number of physical partitions (PPs) but it will be neglectable comparing to issues occurring when data starts growing beyond 50-, 100-, 150GB.

Why CustomerId might not be the best partition key for the data sets which are expected to grow beyond 50GB?

Main reason is that Cosmos DB is designed to scale horizontally and provisioned throughput per PP is limited to the [total provisioned per container (or DB)] / [number of PP].

Once PP split occurs due to exceeding 50GB size your max throughput for existing PPs as well as two newly created PPs will be lower then it was before split.

So imagine following scenario (consider days as a measure of time between actions):

  1. You've created container with provisioned 10k RUs and CustomerId partition key (which will generate one underlying PP1). Maximum throughput per PP is 10k/1 = 10k RUs
  2. Gradually adding data to container you end-up with 3 big customers with C1[10GB], C2[20GB] and C3[10GB] of invoices
  3. When another customer was onboarded to the system with C4[15GB] of data Cosmos DB will have to split PP1 data into two newly created PP2 (30GB) and PP3 (25GB). Maximum throughput per PP is 10k/2 = 5k RUs
  4. Two more customers C5[10GB] C6[15GB] were added to the system and both ended-up in PP2 which lead to another split -> PP4 (20GB) and PP5 (35GB). Maximum throughput per PP is now 10k/3 = 3.333k RUs

enter image description here

IMPORTANT: As a result on [Day 2] C1 data was queried with up to 10k RUs but on [Day 4] with only max to 3.333k RUs which directly impacts execution time of your query

This is a main thing to remember when designing partition keys in current version of Cosmos DB (12.03.21).

user_9658410
Related