Cosmos DB Change Feed with Azure Functions

1. Understanding the Cosmos DB Change Feed

At its core, the Cosmos DB change feed is a persistent, append-only log of all the changes (inserts and updates) that occur in a container, in the order they happen. It's not a push notification system; rather, it's a log that can be read by one or more consumers.

Azure Functions provide the simplest and most common way to consume this feed, using a special trigger that handles all the complexity for you.

To get a concise overview of what the change feed is, its key features, and its common use cases, please watch the beginning of the following video.

As the video highlighted, you don't need to enable the change feed; it's always on. The key challenge lies in how to consume it reliably. This is where the concept of a "lease container" comes in.

2. The Role of the Lease Container

When you use an Azure Function to process the change feed, it needs to keep track of its progress. Which changes has it already processed? This prevents it from reprocessing the same changes after a restart and allows multiple function instances to work in parallel on the same change feed without stepping on each other's toes.

This state is stored in a separate container called the lease container.

• Checkpointing: The function periodically writes a "checkpoint" to the lease container, marking the last processed change.
• Parallelism: The change feed is processed per partition. The lease container manages the distribution of these partitions across different instances of your function, enabling you to scale out automatically.

The architecture looks like this:

You must have a lease container for your trigger to work. Fortunately, as you'll see, the Azure Function trigger can be configured to create this for you automatically.

3. Configuring the CosmosDBTrigger in C#

Your existing experience with C# and Azure Functions will be very helpful here. To connect an Azure Function to the change feed, you use the CosmosDBTrigger attribute.

Let's look at the essential properties you'll need to configure:

[FunctionName("MyChangeFeedProcessor")]
public static void Run(
    [CosmosDBTrigger(
        databaseName: "TasksDB",
        containerName: "Items",
        Connection = "CosmosDbConnectionString",
        LeaseContainerName = "leases",
        CreateLeaseContainerIfNotExists = true)] IReadOnlyList<ToDoItem> input,
    ILogger log)
{
    // ... processing logic
}

Let's break down these attributes:

• databaseName: The name of the database containing the container you want to monitor.
• containerName: The name of the container whose changes you want to process.
• Connection: The name of the application setting in local.settings.json (or your Function App's configuration) that holds the Cosmos DB connection string.
• LeaseContainerName: The name of the container to use for storing leases. A common convention is simply leases.
• CreateLeaseContainerIfNotExists: A boolean that, when set to true, directs the Functions runtime to automatically create the lease container if it doesn't already exist. This is incredibly convenient for development and initial deployment.

The function receives an IReadOnlyList<T>, where T can be a generic Document or, more usefully, a strongly-typed POCO class that matches your data model. This means your function receives changes in batches, which is more efficient.

For a comprehensive list of all available configuration options, you can refer to the official documentation.

4. Hands-On: Building a Materialized View Processor

Now, let's put this into practice with a common, real-world scenario: creating a materialized view. A materialized view is a pre-calculated, denormalized copy of your data, optimized for a specific query pattern.

Our Goal:

1. We'll have a primary container named books, partitioned by id. This is optimized for writing and looking up books by their unique identifier.
2. We'll create a change feed processor that listens to the books container.
3. When a book is added or updated, the function will write a copy of it to a second container, books-by-genre, partitioned by genre. This new container is optimized for efficiently querying all books of a specific genre.

This pattern allows you to avoid costly cross-partition queries on the primary container when you need to find books by genre.

The following video segments will guide you through building this exact scenario in Visual Studio.

Step 1: Create the Function and Configure the Trigger

First, we'll set up the new Azure Function project, create a Cosmos DB-triggered function, and configure its attributes.

Here's a summary of the trigger configuration you'll be setting up:

[FunctionName("InsertBookByGenre")]
public async Task Run(
    [CosmosDBTrigger(
        databaseName: "BooksDB",
        containerName: "books",
        Connection = "CosmosDbConnectionString",
        LeaseContainerName = "leases",
        CreateLeaseContainerIfNotExists = true)]IReadOnlyList<Document> input,
    ILogger log)
{
    // ...
}

Step 2: Implement the Processing Logic

Next, we'll write the code inside the function to process the incoming batch of documents. The logic is simple: iterate through the documents, deserialize each one into a Book object, and then use a data access method to insert it into the books-by-genre container.

Your processing logic will look something like this:

if (input != null && input.Count > 0)
{
    log.LogInformation($"Documents modified: {input.Count}");

    foreach (var doc in input)
    {
        // The input is a Document object, get its string representation
        string json = doc.ToString(); 
        
        // Deserialize to our Book model
        Book book = JsonConvert.DeserializeObject<Book>(json);
        
        // The video uses a repository pattern, which is great. 
        // For simplicity, you could also use a CosmosClient instance here.
        // await _bookRepository.InsertBookByGenreAsync(book);

        log.LogInformation($"Processed Book Id: {book.Id}");
    }
}

Step 3: Run and Test Locally

Finally, let's run the function and test it. You'll start the function locally, then use the Azure Portal or another tool to add a document to the books container. You should see your function's logs light up in the console, and then you can verify that the new document appears in both the leases container and the books-by-genre container.

By completing these steps, you will have successfully implemented a robust, event-driven processor that responds to data changes in real-time.

Conclusion

Congratulations on completing the Cosmos DB module! You have now mastered a critical serverless pattern for building highly responsive and scalable applications on Azure.

Key Takeaways:

• The Cosmos DB Change Feed provides a persistent log of inserts and updates to a container.
• Azure Functions with the CosmosDBTrigger are the simplest and most robust way to consume the change feed.
• The lease container is a critical component that manages state (checkpointing) and enables parallel processing across multiple function instances.
• A common and powerful use case for the change feed is creating materialized views to optimize read patterns and avoid costly cross-partition queries.

Next Steps

We are now moving from the core data storage services to the equally important topic of security. In the next module, Implementing Secure Cloud Solutions, we will begin by exploring the foundational components of identity and access management in Azure. Our first lesson will cover the Microsoft Identity Platform, including Microsoft Entra ID (formerly Azure Active Directory), app registrations, and the core authentication protocols OAuth 2.0 and OpenID Connect.