Structured Logging for Production

Why Plain Text Fails in Production

A human can easily read a multi-line log entry, but for a machine, it's ambiguous. Where does one log message end and the next begin? How do you reliably extract the timestamp or the severity level if the format changes slightly? This problem becomes critical when dealing with exceptions.

To see why, let's watch a short clip that illustrates the problem and introduces the solution: logging in a structured format like JSON.

As the video showed, by formatting each log entry as a single line of valid JSON, we create a record that is unambiguous and easy for any system to parse.

Implementing JSON Logging with python-json-logger

While you could write your own JsonFormatter class from scratch (as demonstrated later in the mCoding video), the standard and most robust approach is to use a dedicated, well-tested library. The community standard is python-json-logger.

First, you'll need to install it:

pip install python-json-logger

This library provides a JsonFormatter that you can drop into your logging configuration. It seamlessly replaces the standard logging.Formatter.

The core idea is simple: instead of creating a logging.Formatter, you instantiate a jsonlogger.JsonFormatter.

Centralizing Configuration with dictConfig

In the previous lesson, we configured our logger programmatically. While useful for understanding the components, this approach mixes configuration with application logic. For production systems, the best practice is to use logging.config.dictConfig to load the entire logging setup from a dictionary, which is typically stored in a separate YAML or JSON file.

This separates concerns and allows you to change logging behavior (e.g., verbosity, output format) without modifying your application code, which is essential for managing different environments (dev, staging, prod).

Let's look at how to integrate python-json-logger into a dictConfig setup.

Here's a self-contained example showing a dictConfig that defines two handlers: one for human-readable console output and one for machine-readable JSON file output.

import logging
import logging.config
import yaml




# --- config.yaml ---
# (In a real app, this would be a separate file)
LOGGING_CONFIG_YAML = """
version: 1
disable_existing_loggers: false

formatters:
  simple:
    format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
  json:
    '()': 'pythonjsonlogger.jsonlogger.JsonFormatter'
    format: "%(asctime)s %(levelname)s %(name)s %(message)s"

handlers:
  console:
    class: logging.StreamHandler
    level: INFO
    formatter: simple
    stream: ext://sys.stdout
  json_file:
    class: logging.FileHandler
    level: WARNING
    formatter: json
    filename: app_structured.log

loggers:
  my_app:
    level: INFO
    handlers: [console, json_file]
    propagate: false

root:
  level: WARNING
  handlers: [console]
"""




# Load the config from the YAML string
config = yaml.safe_load(LOGGING_CONFIG_YAML)
logging.config.dictConfig(config)




# Get a logger for our application
logger = logging.getLogger('my_app')




# --- Use the logger ---
logger.info("This is an informational message.")
logger.warning("This is a warning message.")
try:
    1 / 0
except ZeroDivisionError:
    logger.exception("An unhandled exception occurred.")

After running this script:

1. Console output will be human-readable:

   2023-11-20 14:30:00,123 - my_app - INFO - This is an informational message.
   2023-11-20 14:30:00,123 - my_app - WARNING - This is a warning message.
   2023-11-20 14:30:00,124 - my_app - ERROR - An unhandled exception occurred.
   Traceback (most recent call last):
     File "test.py", line 47, in <module>
       1 / 0
   ZeroDivisionError: division by zero
   

2. app_structured.log file will contain structured JSON (note that the traceback is a single string field):

   {"asctime": "2023-11-20 14:30:00,123", "levelname": "WARNING", "name": "my_app", "message": "This is a warning message."}
   {"asctime": "2023-11-20 14:30:00,124", "levelname": "ERROR", "name": "my_app", "message": "An unhandled exception occurred.", "exc_info": "Traceback (most recent call last):\n  File \"test.py\", line 47, in <module>\n    1 / 0\nZeroDivisionError: division by zero"}
   

Enriching Logs with Context

A key benefit of structured logging is the ability to add rich, queryable context to your logs. For example, in a web application, you might want every log message generated during a request to include the request_id and user_id.

The extra Parameter

The simplest way to add context is with the extra keyword argument in your logging calls. python-json-logger automatically adds any keys from the extra dictionary to the final JSON output.

Example usage:

logger.info(
    "User attempted to log in.", 
    extra={'user_email': 'test@example.com', 'request_id': 'abc-123'}
)

This would produce a JSON log containing "user_email": "test@example.com" and "request_id": "abc-123".

Automatic Context with Filters and contextvars

Passing extra manually for every log call is tedious and error-prone. A much more powerful, production-grade pattern is to inject context automatically. This is typically done using:

1. A custom logging.Filter to add data to each LogRecord.
2. Python's contextvars module to store context in a way that is safe for concurrent code (like web requests or async tasks).

This pattern allows you to set a request_id once at the beginning of a request, and have it automatically appear in every single log message generated during that request's lifecycle, without any further effort.

An Alternative: structlog

While python-json-logger enhances the standard logging module, some developers prefer a library designed from the ground up for structured logging. structlog is the leading choice in this category.

It uses a pipeline of "processors" to build up a log event dictionary, which gives you explicit control over the final output. For a developer like you who values modern tooling and robust design, it's worth knowing about.

Exercise: Refactor to a dictConfig with JSON Logging

Let's apply these concepts. You will refactor the logging setup from the previous lesson's exercise to use dictConfig and output JSON to a file.

Starter main.py:

import logging
import requests




# Assume this function loads and applies the config
def setup_logging():



    # You will implement this function using dictConfig
    pass

def fetch_user_profile_logged(user_id: int):
    """Fetches a user profile from a public API using logging."""
    logger = logging.getLogger(__name__) # Get logger inside function for this example
    
    logger.info(f"Starting to fetch profile for user {user_id}")
    try:
        response = requests.get(f"https://jsonplaceholder.typicode.com/users/{user_id}")
        response.raise_for_status()
        logger.info(
            f"Successfully fetched profile for user {user_id}",
            extra={'user_id': user_id, 'response_code': response.status_code}
        )
        return response.json()
    except requests.exceptions.RequestException as e:
        logger.exception(
            f"Failed to fetch profile for user {user_id}.",
            extra={'user_id': user_id, 'error_type': type(e).__name__}
        )
        return None




# --- Main execution ---
setup_logging()
fetch_user_profile_logged(1)
fetch_user_profile_logged(999)

Your Task:

1. Make sure you have PyYAML and python-json-logger installed (pip install pyyaml python-json-logger).
2. Create a YAML configuration string (or a separate config.yaml file).
3. The configuration should define two handlers:
   • A console handler that prints INFO and higher logs in a simple, human-readable format.
   • A json_file_handler that writes WARNING and higher logs to a file named errors.jsonl in JSON format.
4. Implement the setup_logging() function to load this YAML configuration and apply it using logging.config.dictConfig().
5. Run the script and verify that:
   • Your console shows the human-readable INFO, WARNING, and ERROR messages.
   • The errors.jsonl file contains only the WARNING and ERROR messages as JSON objects, including the custom fields from the extra dictionary.

import logging
import logging.config
import requests
import yaml




# The YAML configuration as a multiline string
LOGGING_CONFIG_YAML = """
version: 1
disable_existing_loggers: false

formatters:



  # Formatter for human-readable console output
  simple:
    format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
  



  # Formatter for machine-readable JSON output
  json:
    '()': 'pythonjsonlogger.jsonlogger.JsonFormatter'
    format: "%(asctime)s %(levelname)s %(name)s %(message)s %(user_id)s"

handlers:



  # Handler for console output
  console:
    class: logging.StreamHandler
    level: INFO
    formatter: simple
    stream: ext://sys.stdout




  # Handler for JSON file output for warnings and errors
  json_file:
    class: logging.FileHandler
    level: WARNING
    formatter: json
    filename: errors.jsonl  # .jsonl is a common extension for JSON Lines files
    mode: 'w' # Overwrite the file on each run for this example




# Configure the root logger and any specific app loggers
root:
  level: INFO
  handlers: [console, json_file]
"""

def setup_logging():
    """Loads logging configuration from YAML and applies it."""
    config = yaml.safe_load(LOGGING_CONFIG_YAML)
    logging.config.dictConfig(config)
    logging.info("Logging configured from YAML.")

def fetch_user_profile_logged(user_id: int):
    """Fetches a user profile from a public API using logging."""
    logger = logging.getLogger(__name__)
    
    logger.info(f"Starting to fetch profile for user {user_id}", extra={'user_id': user_id})
    try:
        response = requests.get(f"https://jsonplaceholder.typicode.com/users/{user_id}")
        response.raise_for_status()
        logger.info(
            f"Successfully fetched profile for user {user_id}",
            extra={'user_id': user_id, 'response_code': response.status_code}
        )
        return response.json()
    except requests.exceptions.RequestException as e:
        logger.exception(
            f"Failed to fetch profile for user {user_id}.",
            extra={'user_id': user_id, 'error_type': type(e).__name__}
        )
        return None




# --- Main execution ---
setup_logging()
fetch_user_profile_logged(1)
fetch_user_profile_logged(999)

{"asctime": "...", "levelname": "ERROR", "name": "__main__", "message": "Failed to fetch profile for user 999.", "user_id": 999, "exc_info": "...", "error_type": "HTTPError"}

Conclusion

You have now mastered a crucial skill for writing production-grade Python applications. By moving from human-readable text to machine-readable JSON, your logs are no longer just diagnostic messages; they are a rich, queryable dataset that can power monitoring, alerting, and deep analysis.

Key Takeaways:

• Structured logs are for machines: Use JSON for any logs that will be parsed by another system.
• Use python-json-logger: It's the standard, robust library for this task.
• Centralize with dictConfig: Define your logging configuration declaratively in a separate file (like config.yaml) to keep it separate from application logic.
• Enrich with context: Use the extra parameter for one-off context, and the Filter + contextvars pattern for automatically injecting request-scoped data.
• Consider structlog: For new projects where structured logging is a primary concern, structlog offers a powerful and clean alternative.

In this lesson and the previous ones, we've focused on how to handle and record errors. But when should you raise an exception versus using a simple check? In our next lesson, we will explore this by tackling the learning outcome: Distinguish between assert for internal sanity checks during development and exceptions for validating external inputs.