Number of records based on Partition key - Is there a max/min

user_8427453 0
Viewed 284

We have some 20 million documents which needs to be inserted into our Cosmos DB. Our partition ID for these documents is based on Group_ID. There can be max 300 unique Person_ID under this Group_ID.

This translates to something like below.

20 million individuals. Each one has a unique PERSON_ID Partitioned by GROUP_ID -- Ranges from 50 to 300

If I consider 300, then roughly we get 66K partitions.

  • Search by GROUP_ID will be our most used ID.

Question : Is it ok to have 66k partitions ? Will there be a performance impact ? Also, size of our Cosmos DB will grow from 20 million to 100 million in 3 years (which is 3.3 million partitions)

azure-cosmosdb
1 Answers

A partition key in Cosmos DB defines a logical partition, which is different than a physical partition. It's useful to read the partitioning docs to understand this difference. A relevant highlight:

There is no limit to the number of logical partitions in your container

Indeed, it's recommended to have as many partitions as possible, assuming they evenly distribute throughput and fit your query patterns. Given that the logical partition size can reach 20 GB and the physical 50 GB, you may not have more than a single physical partition even with a large number of documents across many partition keys.

Performance wise, as long as your data is well distributed by partition keys, the more important thing to focus on is your queries. In particular, the design should minimize the times you need to perform cross partition queries and maximize the times you're able to perform point reads.

user_24117130
Related