Structured Logging in Python

1. From Unstructured Text to Structured Data

At its core, logging is the act of recording events that occur while your application is running. The simplest form of this is something you've likely done countless times:

print("User tried to log in.")

This is an example of unstructured logging. It's a free-form string of text, easy for a human to write and read in a simple console. However, in a production system handling thousands of requests per second, these text streams become a noisy, unsearchable "wall of text."

Now, consider this alternative:

{
  "timestamp": "2023-10-27T10:00:00Z",
  "level": "INFO",
  "message": "User login attempt.",
  "event": "user.login",
  "user_id": "usr_a1b2c3d4",
  "source_ip": "192.168.1.100"
}

This is structured logging. Each piece of information is a key-value pair, making the log entry machine-readable, like a row in a database. This fundamental shift from a simple string to a data object unlocks immense power.

To understand the benefits of this shift, let's hear a quick explanation.

As the video explains, the primary benefits of structured logging are:

• Search and Filtering: You can run precise queries like level=ERROR AND user_id="usr_a1b2c3d4".
• Analysis and Visualization: You can create dashboards to track error rates, login failures per IP, and other key business or operational metrics.
• Automated Alerting: You can set up automated alerts for specific conditions, such as "alert if the rate of 500 errors exceeds 10 per minute."

Now that you understand why structured logging is essential, let's explore how to implement it effectively in Python.

2. Python's logging Module: The Machinery

Python comes with a powerful and flexible built-in logging module. While it can seem complex, understanding its core components is crucial for building a robust logging system. It's much more than a glorified print function.

The key components form a processing pipeline for every log event:

1. Loggers: Your entry point into the system. You get a logger instance in your code (e.g., logging.getLogger(__name__)) and call methods on it (.info(), .error(), etc.). This creates a LogRecord object.
2. Handlers: These determine where the LogRecord is sent. To the console (StreamHandler), a file (FileHandler), over the network, etc. A logger can have multiple handlers.
3. Formatters: These take the LogRecord object and serialize it into a string. This is where the magic of structured logging happens—you use a formatter that outputs JSON.
4. Filters: These provide fine-grained control to decide if a LogRecord should be processed or to enrich it with additional data before it reaches the formatter.

The following video provides an excellent mental model for how these pieces fit together and introduces the modern best practice for configuring them.

3. Implementing Structured Logging with dictConfig

As you just saw, the most robust way to set up logging is declaratively using logging.config.dictConfig. This approach is clean, centralized, and allows you to easily switch configurations between different environments (e.g., development vs. production).

Let's put theory into practice. We will create a configuration that directs all logs to the console as structured JSON. While you could write your own JSON formatter (as shown later in the mCoding video), a common practice is to use a well-supported library like python-json-logger.

First, install the library:

pip install python-json-logger

Next, create a configuration file, for example, logging_config.yaml:

version: 1
disable_existing_loggers: False

formatters:
  json:
    (): pythonjsonlogger.jsonlogger.JsonFormatter
    format: "%(asctime)s %(name)s %(levelname)s %(message)s"
    rename_fields:
      levelname: level
      asctime: time

handlers:
  console:
    class: logging.StreamHandler
    formatter: json
    stream: ext://sys.stdout

root:
  level: INFO
  handlers: [console]

loggers:



  # Set a more verbose level for our specific application's logger
  my_app:
    level: DEBUG
    handlers: [console]
    propagate: False # Prevents logs from being passed to the root logger and duplicated

This configuration does the following:

• Defines a json formatter using python-json-logger.
• Defines a console handler that writes to standard output using the json formatter.
• Configures the root logger to handle logs from all libraries at INFO level and above.
• Configures a specific logger named my_app to be more verbose (DEBUG) and ensures its logs are not duplicated by the root logger (propagate: False).

To use this configuration, you load it at your application's entry point. You'll also need PyYAML to parse the file.

pip install PyYAML

# main.py
import logging
import logging.config
import yaml




# Load the logging configuration
with open("logging_config.yaml", "r") as f:
    config = yaml.safe_load(f.read())
    logging.config.dictConfig(config)




# Get a logger for this specific module
# Using __name__ is a best practice as it creates a hierarchical logger name
# e.g., 'my_app.services.users'
logger = logging.getLogger("my_app.main")

def main():
    logger.debug("Application starting up.")
    logger.info("This is an informational message.")
    logger.warning("Something might be wrong here.")
    try:
        1 / 0
    except ZeroDivisionError:
        logger.error("A division by zero occurred.", exc_info=True)

    logger.debug("Application shutting down.")

if __name__ == "__main__":
    main()

When you run this script, you will see structured JSON output for each log call, including a formatted traceback for the exception.

{"time": "...", "name": "my_app.main", "level": "DEBUG", "message": "Application starting up."}
{"time": "...", "name": "my_app.main", "level": "INFO", "message": "This is an informational message."}
{"time": "...", "name": "my_app.main", "level": "WARNING", "message": "Something might be wrong here."}
{"time": "...", "name": "my_app.main", "level": "ERROR", "message": "A division by zero occurred.", "exc_info": "..."}
{"time": "...", "name": "my_app.main", "level": "DEBUG", "Application shutting down."}

This approach provides a solid foundation. For a deeper look into all the components and patterns for professional logging, the following guide is an excellent, comprehensive resource.

4. The Power of Contextual Logging

Having structured logs is a great start, but their true power is unlocked when you enrich them with context. In a backend service, you want every log message related to a specific user request to be tagged with the same request_id. This allows you to trace the entire journey of a request through your system.

Manually adding this to every log call is tedious and error-prone:

logger.info("Processing data.", extra={"request_id": "xyz-123"})
logger.info("Saving to database.", extra={"request_id": "xyz-123"})

A much more powerful pattern uses Filters and Python's contextvars to automatically inject this context into every log record within a specific scope. This is an advanced but essential technique for production-grade services.

The Dash0 guide you just reviewed has a superb section on this.

By using a middleware in your Django or FastAPI application to wrap each request in with add_to_log_context(request_id=...):, every single log call made during the processing of that request will automatically be enriched with the request_id, without any changes to your business logic code. This is the key to making your logs traceable and powerful for debugging.

Conclusion

In this lesson, we transitioned from thinking about deployment to considering observability. You've learned that effective logging is not about printing text to a console; it's about producing a structured, queryable stream of events that tells the story of your application's behavior.

Key Takeaways:

• Structured logs (e.g., JSON) are for machines. They enable powerful searching, filtering, and analysis that is impossible with unstructured text.
• Python's built-in logging module is a powerful system composed of Loggers, Handlers, Formatters, and Filters.
• The modern best practice is to use logging.config.dictConfig to declaratively set up your logging pipeline, often from a YAML or JSON file.
• You can produce structured logs by using a custom JSON Formatter, often from a library like python-json-logger.
• Contextual logging is the practice of enriching logs with dynamic data (like a request_id) to make them traceable. This can be implemented elegantly using contextvars and a logging Filter.

Preview of the Next Lesson

Logs are fantastic for recording discrete events—what happened. But what about the overall health of your system? How fast are your endpoints, on average? What is the error rate per minute? To answer these questions, we need the second pillar of observability: metrics.

In the next lesson, we will learn how to identify and monitor key backend metrics using the RED method (Rate, Errors, Duration). This will give you a high-level, numerical dashboard view of your application's health, complementing the detailed, event-specific view that logs provide.