Cosmos DB Partition Keys: Selection and Best Practices

Hello! Welcome back to our module on developing solutions with Azure Cosmos DB.

In our last lesson, we introduced the fundamental concepts of Cosmos DB, including its resource model. We established that the container is the unit of scalability. Today, we're going to dive into arguably the most critical design decision you'll make when working with Cosmos DB: the partition key. This is the how behind a container's ability to scale.

Our learning outcome for this lesson is to: Explain the role of partition keys in Cosmos DB and best practices for selection. Getting this right is crucial for building a high-performance, cost-effective application, and it's a frequent topic on the AZ-204 exam.

1. Understanding Partitioning: Logical vs. Physical

At a high level, partitioning is the mechanism Cosmos DB uses to divide your container's data and throughput into smaller, more manageable pieces, allowing it to scale out horizontally across many servers. To understand this, we need to differentiate between two types of partitions:

Logical Partitions: A logical partition is a set of items that all share the same partition key value. For example, if you partition a Users container by country, all users from 'Canada' would reside in a single logical partition. You, as the developer, primarily think in terms of logical partitions. A key constraint to remember is that a single logical partition cannot exceed 20 GB of data.
Physical Partitions: A physical partition is an internal implementation detail managed entirely by Azure. It's a reserved amount of SSD storage and compute that provides up to 10,000 Request Units per second (RU/s) and can store up to 50 GB of data. A single physical partition can host one or many logical partitions.

When you write an item to a container, Cosmos DB uses the partition key value to hash it and determine which logical partition it belongs to. Azure then transparently manages the mapping of these logical partitions to physical partitions to handle the load.

Azure Cosmos DB Logical and Physical Partitioning — This diagram from Microsoft shows how a container's logical partitions (defined by a partition key like 'city') are mapped to underlying physical partitions. These physical partitions are replicated across geographic regions to provide global distribution and high availability.

To see these concepts in action, let's watch the first part of the following video.

What is Partitioning in Azure Cosmos DB?

This video from Will Velida provides a clear explanation of partitioning, the difference between logical and physical partitions, and the problem of 'hot partitions'.

Watch from the beginning of the discussion on partitioning at 01:26 until 06:27. Focus on understanding how logical partitions are formed and how they map to physical partitions, as well as the concept of a 'hot partition'.

As the video explained, the goal is to have your data and requests spread evenly across many logical partitions. When one logical partition receives a disproportionate amount of requests or stores a disproportionate amount of data, it becomes a hot partition. This leads to inefficient use of your provisioned throughput, potential request throttling, and higher costs, even if other partitions are idle.

2. Best Practices for Choosing a Partition Key

Choosing your partition key is a permanent decision for a container. If you get it wrong, the only way to fix it is to create a new container and migrate all your data. Therefore, it's vital to choose wisely based on a few core principles.

A good partition key has three main characteristics:

High Cardinality: The property you choose should have a very wide range of possible values.
- Good example: userId in a system with millions of users. This creates millions of small logical partitions.
- Bad example: A boolean isActive property. This would create only two logical partitions ('true' and 'false'), forcing all your data into just two buckets.
Even Distribution: The key should distribute request traffic (RUs) and data storage evenly across all logical partitions.
- Good example: productId in an e-commerce catalog where products have a similar amount of data and access patterns.
- Bad example: city for a customer base where 90% of customers are in 'New York'. The 'New York' partition would become a massive hot spot for both data and requests.
Aligns with Query Patterns: Your most frequent and performance-critical queries should include an equality filter on the partition key (e.g., WHERE c.userId = 'some-user-id').
- In-partition query: When you provide the partition key in your query, Cosmos DB can efficiently route the request directly to the single physical partition that holds the data. This is fast and cheap in terms of RUs.
- Cross-partition query: If you don't provide the partition key, Cosmos DB must "fan out" the query to every physical partition and aggregate the results. This is significantly slower and more expensive.

To explore these design principles further, please watch the following video from the Azure team.

Partition Strategy | Azure Cosmos DB Essentials Season 2

This 'Cosmos DB Essentials' video clearly articulates the design principles for choosing a partition key based on your application's workload (read-heavy vs. write-heavy).

Watch the section 'Design principles' from 04:53 to 07:25. Pay close attention to the IoT example and how it illustrates choosing a key for a write-heavy vs. a read-heavy workload.

3. Partitioning Strategies

Based on these principles, several common strategies have emerged.

Partitioning and horizontal scaling - Azure Cosmos DB

The official Microsoft documentation provides the definitive guide on choosing a partition key. This section formalizes the principles we've discussed.

Read the sections titled 'Choose a partition key' and 'Partition keys for read-heavy containers'. The first section details the essential properties of a good key, and the second explains how to optimize for query patterns.

Let's summarize the main strategies from the documentation and common practice:

Strategy	When to Use	Example
Natural Key	Your data has a natural property that has high cardinality and is used in most queries. This is the ideal scenario.	In an e-commerce app, partition the `Orders` container by `/customerId`. Most queries will be to get orders for a specific customer.
Item ID (`/id`) as Key	Best for write-heavy workloads where you need to spread writes as evenly as possible, or for small read-heavy containers. Point reads (fetching an item by its ID) are extremely efficient.	An event logging system where each event has a unique ID. Partitioning by `/id` guarantees perfect write distribution.
Synthetic Key	No single property satisfies all criteria. You create a new property by combining two or more existing ones.	In an IoT application, a device might send data every second. Partitioning by `/deviceId` could create a hot spot. A synthetic key like `/partitionKey` with a value of `"{deviceId}:{yyyy-MM-dd}"` spreads the data for a single device across multiple days.
Hierarchical Key	For very large, complex datasets where a single key isn't enough. Allows up to three levels of partitioning.	A multi-tenant application could use `/tenantId`, `/userId` as hierarchical keys to avoid the 20 GB limit for a single large tenant.

4. Common Mistakes to Avoid

Partitioning by Date/Time: This is a classic anti-pattern. All current writes will go to the same logical partition (e.g., for the current hour or day), creating a massive write hot spot.
Using a Low-Cardinality Property: Avoid properties like status, category, or type which have a limited set of values.
Ignoring the 20 GB Logical Partition Limit: If you partition by tenantId and a single tenant could grow to have more than 20 GB of data, your writes will eventually fail for that tenant.
Forgetting about Reads: A key that perfectly distributes writes but forces all your read queries to be cross-partition queries is often a poor choice for overall performance and cost.

Conclusion

We've covered the most important concepts for designing a scalable Cosmos DB container. Understanding partitioning is non-negotiable for success with Cosmos DB and a key area for the AZ-204 exam.

Key Takeaways:

Cosmos DB scales by distributing data across logical partitions (based on your key) which are mapped to physical partitions (managed by Azure).
A good partition key must have high cardinality, ensure even distribution of data and requests, and align with your primary query patterns.
Queries that include the partition key (in-partition) are fast and cheap; queries that don't (cross-partition) are slow and expensive.
Avoid anti-patterns like partitioning by date or low-cardinality fields, and always be mindful of the 20 GB logical partition size limit.
The choice of partition key is immutable; it cannot be changed after the container is created without a data migration.

Next Steps

Now that we have a strong theoretical understanding of partitioning, it's time to put it into practice. In our next lesson, we will tackle the learning outcome: "Create a Cosmos DB account, database, and container with an appropriate partition key." This will be a hands-on lab where you'll provision your first Cosmos DB resources and apply the principles learned today.