Structured Logging in Python: JSON and Fields

Unstructured text logs like "Order failed: timeout after 30s" are hard to search and aggregate across thousands of instances. Structured logging emits each log as a JSON object with named fields, so log aggregation systems (Datadog, Elasticsearch, Splunk, CloudWatch) can parse fields like user_id, duration_ms, and error_code without regex parsing. This transforms logs from a human read-only record into a queryable dataset that powers alerts, dashboards, and automated incident response.

Structured logging is not an all-or-nothing proposition. You start by adding context fields to a standard Python logger, then optionally use a dedicated library (structlog, python-json-logger) to emit pure JSON. The benefit is enormous: instead of searching logs for the string "timeout", you can query event_type = "timeout" AND duration_ms > 30000, which is both faster and more reliable.

Why Is Structured Logging Better Than Text Logs?

Text logs like "ERROR: Database connection failed after 5 retries" are optimized for human reading but terrible for machines. Log aggregation systems must either fail silently or apply brittle regex to extract meaning. Structured logs encode the same information as fields, so any system downstream can understand and act on them.

Consider two log lines for the same event:

Unstructured: 2026-06-02 14:23:45,789 - myapp.database - ERROR - Query execution failed: timeout after 5s on server db-prod-1

Structured (JSON): {"timestamp": "2026-06-02T14:23:45.789Z", "logger": "myapp.database", "level": "ERROR", "message": "Query execution failed", "error_type": "timeout", "duration_ms": 5000, "server": "db-prod-1"}

With structured logs, you can:

• Alert when any query takes more than 3000 ms: duration_ms > 3000
• Count failures per server: error_type = "timeout" | stats count by server
• Build a histogram of query durations: logger = "myapp.database" | stats percentile(duration_ms, 95)

Unstructured logs require humans to read and manually hunt for patterns.

How Do You Add Structured Fields to Standard Python Logging?

The logging module's LogRecord object has an __dict__ that you can extend with custom fields. The standard way is to use the extra parameter when logging:

import logging
import json

logger = logging.getLogger(__name__)

def process_order(order_id, user_id):
    logger.info(
        "Order processing started",
        extra={
            'order_id': order_id,
            'user_id': user_id,
            'timestamp_ms': int(time.time() * 1000)
        }
    )
    
    try:
        result = charge_card(order_id)
    except TimeoutError as e:
        logger.warning(
            "Card charge timed out, will retry",
            extra={
                'order_id': order_id,
                'error_type': 'timeout',
                'retry_count': 1,
                'duration_ms': 5000
            }
        )
    
    logger.info(
        "Order processing completed",
        extra={
            'order_id': order_id,
            'status': 'success'
        }
    )

These extra fields are added to the LogRecord but still appear as text in unstructured logs. To emit actual JSON, you need a custom formatter.

How Do You Emit Logs as JSON?

The simplest approach is to write a custom formatter that converts the entire LogRecord to JSON:

import logging
import json
import sys

class JSONFormatter(logging.Formatter):
    def format(self, record):
        log_obj = {
            'timestamp': self.formatTime(record, self.datefmt),
            'level': record.levelname,
            'logger': record.name,
            'message': record.getMessage(),
        }
        # Add any extra fields
        if hasattr(record, '__dict__'):
            for key, value in record.__dict__.items():
                if key not in ('name', 'msg', 'args', 'created', 'filename', 
                              'funcName', 'levelname', 'levelno', 'lineno', 
                              'module', 'msecs', 'message', 'pathname', 'process',
                              'processName', 'relativeCreated', 'thread', 'threadName'):
                    log_obj[key] = value
        return json.dumps(log_obj)

# Configure the root logger
logger = logging.getLogger()
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JSONFormatter())
logger.addHandler(handler)

# Now emit JSON logs
logger.info("Application started", extra={'version': '1.2.3', 'environment': 'prod'})
# Output: {"timestamp": "2026-06-02 14:23:45,789", "level": "INFO", "logger": "root", 
#          "message": "Application started", "version": "1.2.3", "environment": "prod"}

This is the foundation. For more complex scenarios, libraries like python-json-logger provide pre-built formatters:

import logging
from pythonjsonlogger import jsonlogger

logger = logging.getLogger()
handler = logging.StreamHandler()
formatter = jsonlogger.JsonFormatter()
handler.setFormatter(formatter)
logger.addHandler(handler)

logger.info("Payment processed", extra={'user_id': 42, 'amount': 99.99})
# Output: {"message": "Payment processed", "user_id": 42, "amount": 99.99, "timestamp": "2026-06-02 14:23:45,789"}

What Is the Difference Between Standard Logging and structlog?

The standard logging module with JSON formatters works well for most applications. The structlog library (a third-party package) takes a different approach: instead of passing extra dictionaries, you bind context to a logger instance, and all subsequent logs from that instance include those fields automatically.

# Standard logging with extra
import logging
logger = logging.getLogger(__name__)

logger.info("Request received", extra={'request_id': 'abc123', 'user_id': 42})
logger.info("Processing", extra={'request_id': 'abc123', 'user_id': 42})
logger.info("Response sent", extra={'request_id': 'abc123', 'user_id': 42})
# Must repeat request_id and user_id every time

# structlog approach
import structlog
logger = structlog.get_logger(__name__)

logger = logger.bind(request_id='abc123', user_id=42)
logger.info("Request received")  # request_id and user_id are automatically included
logger.info("Processing")         # request_id and user_id are automatically included
logger.info("Response sent")      # request_id and user_id are automatically included

structlog is more ergonomic for request-scoped context (like trace IDs), but it adds a dependency. The standard library with JSON formatters is simpler and sufficient for most applications.

How Do You Handle Exceptions in Structured Logs?

When logging exceptions, include the error type, message, and optionally the stack trace as fields:

import logging
import traceback
import sys

logger = logging.getLogger(__name__)

def risky_operation():
    try:
        result = do_something()
    except ValueError as e:
        exc_info = sys.exc_info()
        logger.error(
            "Validation error",
            extra={
                'error_type': type(e).__name__,
                'error_message': str(e),
                'stack_trace': traceback.format_exc()
            }
        )
    except Exception as e:
        logger.critical(
            "Unexpected error",
            extra={
                'error_type': type(e).__name__,
                'error_message': str(e),
                'stack_trace': traceback.format_exc()
            }
        )

Alternatively, use the exc_info parameter, which the formatter will extract:

import logging
logger = logging.getLogger(__name__)

try:
    result = risky_operation()
except Exception as e:
    logger.exception("Operation failed")  # exc_info is set automatically
    # Output: {"message": "Operation failed", "exc_info": "Traceback..."}

Key Takeaways

• Structured logging converts logs to queryable JSON with named fields.
• Use the extra parameter to pass fields to standard logging.
• Write a custom JSONFormatter or use python-json-logger to emit JSON.
• Include error types, user IDs, request IDs, and durations as fields.
• Structured logs enable alerts, dashboards, and automated responses.

Frequently Asked Questions

Should I log every field that might be relevant?

No. Log fields that operators or systems will query on. Too many fields pollute the index and slow down analysis. Typical fields: request_id, user_id, duration_ms, error_type, server, environment.

How do I avoid logging sensitive data like passwords?

Never log passwords, tokens, or PII in extra fields. If you need to include a user identifier, use a hashed ID or user UUID, not email or phone number. Use filters to redact secrets.

Can I use structured logging for custom metrics?

You can, but metrics are better sent to a time-series database (Prometheus, CloudWatch) rather than aggregated from logs. Logs are for events; metrics are for numeric values over time.

What if the log aggregation system does not support JSON?

Most modern systems (Datadog, Splunk, CloudWatch) auto-detect JSON and parse it. If using an older system, JSON is still valid text and human-readable, just not automatically parsed.

How do I correlate logs from different services?

Use a trace ID or request ID field that flows through all services. Instrumentation libraries like OpenTelemetry propagate this automatically; with manual logging, pass it through function parameters or context variables.

Further Reading

• python-json-logger library
• structlog: structured logging for Python
• Google Cloud: Structured logging best practices
• ELK Stack: Parsing and analyzing JSON logs