Python Handlers and Formatters: Route and Format Logs
Handlers and formatters are the two mechanisms that control where logs go and how they appear. A handler is a destination (file, console, network socket, HTTP endpoint). A formatter is a template that transforms a LogRecord into a string. Decoupling these two allows you to send the same log message to a file in one format and to a remote server in another format, without duplicating logging statements in your code. This architecture is the reason the logging module scales from tiny scripts to large microservice platforms.
Understanding handlers and formatters is essential for building production-grade observability. You might emit DEBUG logs to a file for local debugging, INFO and above to a console for developers, WARNING and above to a remote syslog server for centralized aggregation, and ERROR and above to a HTTP sink for immediate alerting. All of this is configured once at startup and requires no changes to your application code.
How Do You Route Logs to Different Destinations?
Each logger can have zero or more handlers. When you call logger.info(), the message is passed to every attached handler. Each handler independently decides whether to emit the message (based on its level) and where to send it.
import logging
# Create a logger
logger = logging.getLogger('myapp')
logger.setLevel(logging.DEBUG) # Logger passes all messages
# Handler 1: File handler, accepts DEBUG and above
file_handler = logging.FileHandler('app_debug.log')
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(logging.Formatter('%(asctime)s [%(levelname)s] %(message)s'))
logger.addHandler(file_handler)
# Handler 2: Console handler, accepts INFO and above
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.INFO)
console_handler.setFormatter(logging.Formatter('[%(levelname)s] %(message)s'))
logger.addHandler(console_handler)
# Handler 3: File handler for errors only
error_handler = logging.FileHandler('app_error.log')
error_handler.setLevel(logging.ERROR)
error_handler.setFormatter(logging.Formatter('ERROR at %(asctime)s: %(message)s'))
logger.addHandler(error_handler)
# Now logs route to all three handlers according to their levels
logger.debug("Debug message") # Goes to file_handler only
logger.info("Info message") # Goes to file_handler and console_handler
logger.error("Error message") # Goes to all three handlers
This pattern is powerful: different stakeholders see different subsets of logs. Developers see everything on console; ops teams get a complete audit trail in one file; alerts monitor the error file.
What Are the Built-In Handlers?
Python provides a rich set of handlers for common destinations:
| Handler | Destination | Typical Use |
|---|---|---|
| StreamHandler | stdout/stderr | Local development, console output |
| FileHandler | File on disk | Audit trail, permanent record |
| RotatingFileHandler | Rotated log files by size | Long-running apps, prevent disk fill |
| TimedRotatingFileHandler | Rotated by time (daily, hourly) | Time-based log organization |
| SysLogHandler | Syslog daemon (RFC 5424) | Centralized Unix/Linux logging |
| HTTPHandler | HTTP POST to endpoint | Remote log aggregation |
| NTEventLogHandler | Windows Event Log | Windows-native logging |
| SMTPHandler | Email (sends on ERROR/CRITICAL) | Critical alerts |
Most applications use a combination of StreamHandler (development), FileHandler or RotatingFileHandler (audit trail), and either SysLogHandler or HTTPHandler (centralized ops).
import logging
from logging.handlers import RotatingFileHandler, SysLogHandler
logger = logging.getLogger('myapp')
logger.setLevel(logging.DEBUG)
# Rotating file: max 10 MB per file, keep 5 backups
rotating = RotatingFileHandler(
'app.log',
maxBytes=10 * 1024 * 1024,
backupCount=5
)
rotating.setLevel(logging.DEBUG)
logger.addHandler(rotating)
# Syslog: send to local syslog daemon
syslog = SysLogHandler(address='/dev/log')
syslog.setLevel(logging.WARNING)
logger.addHandler(syslog)
logger.info("This goes to app.log (rotated)")
logger.warning("This goes to app.log and syslog")
How Do You Create Custom Handlers?
For destinations not covered by built-in handlers, subclass Handler and override the emit() method:
import logging
import json
import requests
class DatadogHandler(logging.Handler):
"""Send logs to Datadog API."""
def __init__(self, api_key, source='python-app'):
super().__init__()
self.api_key = api_key
self.source = source
self.url = 'https://http-intake.logs.datadoghq.com/v1/input'
def emit(self, record):
try:
log_entry = {
'timestamp': int(record.created * 1000),
'message': self.format(record),
'level': record.levelname,
'logger': record.name,
'source': self.source
}
headers = {'DD-API-KEY': self.api_key, 'Content-Type': 'application/json'}
requests.post(f"{self.url}/{self.api_key}", json=log_entry, headers=headers)
except Exception:
self.handleError(record) # Prevent logging handler from crashing app
# Usage
logger = logging.getLogger('myapp')
datadog = DatadogHandler(api_key='your-api-key')
datadog.setLevel(logging.INFO)
logger.addHandler(datadog)
logger.info("This message is sent to Datadog")
The critical rule: never let a handler exception crash your application. Use handleError() to gracefully degrade.
What Are Formatters and When Should You Use Them?
A formatter converts a LogRecord into a string. The format string uses named placeholders like %(asctime)s, %(levelname)s, %(message)s, and custom fields from the extra dict. Different handlers can use different formatters.
import logging
logger = logging.getLogger('myapp')
# Simple format for console
console_handler = logging.StreamHandler()
console_formatter = logging.Formatter('[%(levelname)s] %(message)s')
console_handler.setFormatter(console_formatter)
# Detailed format for file
file_handler = logging.FileHandler('app.log')
file_formatter = logging.Formatter(
'%(asctime)s - %(name)s - %(levelname)s - %(funcName)s:%(lineno)d - %(message)s'
)
file_handler.setFormatter(file_formatter)
logger.addHandler(console_handler)
logger.addHandler(file_handler)
logger.info("Payment processed")
# Console: [INFO] Payment processed
# File: 2026-06-02 14:23:45,789 - myapp - INFO - process_payment:45 - Payment processed
Common format attributes:
%(asctime)s: timestamp with milliseconds%(name)s: logger name (module path)%(levelname)s: severity (DEBUG, INFO, WARNING, etc.)%(message)s: log message text%(funcName)s: calling function name%(lineno)d: source code line number%(process)d,%(thread)d: process and thread IDs
For structured logging, use a JSON formatter (as covered in the structured logging article). For text logs, balance readability (short format) with debuggability (detailed format).
How Do You Prevent Handler Errors from Crashing Your App?
Logging handlers run in your application thread. If a handler fails (network timeout, disk full, API error), you must prevent that from crashing the application. The logging module provides the handleError() method:
import logging
class SafeRemoteHandler(logging.Handler):
def emit(self, record):
try:
# Try to send log to remote server
self.send_to_server(record)
except Exception as e:
# Log the handler error, but do not raise
self.handleError(record)
def send_to_server(self, record):
# Simulated network call that might fail
import requests
requests.post('https://logs.example.com/api/logs', json={'msg': self.format(record)}, timeout=2)
logger = logging.getLogger('myapp')
safe_handler = SafeRemoteHandler()
logger.addHandler(safe_handler)
# Even if the remote endpoint is down, the app continues running
logger.info("This might fail to send, but app is unaffected")
By default, handleError() writes to sys.stderr. For critical production services, add metrics to track handler failures so you know when centralized logging is broken.
Key Takeaways
- Attach multiple handlers to a logger to send logs to different destinations.
- Each handler has its own level and formatter.
- Use RotatingFileHandler to prevent disk fill in long-running apps.
- Create custom handlers for integrations (Datadog, CloudWatch, HTTP endpoints).
- Never let handler errors crash your application; use handleError() gracefully.
Frequently Asked Questions
Should I use one logger per module or one global logger?
Use one logger per module via getLogger(__name__). This allows operators to enable/disable logging per module at runtime without code changes.
How do I add request-scoped context (like trace ID) to all logs?
Use the LoggerAdapter class or bound context from structlog. LoggerAdapter wraps a logger and injects extra fields:
import logging
class RequestAdapter(logging.LoggerAdapter):
def process(self, msg, kwargs):
return f"[{self.extra['request_id']}] {msg}", kwargs
adapter = RequestAdapter(logger, {'request_id': 'abc123'})
adapter.info("Processing") # Output includes request_id
How often should I rotate logs if using RotatingFileHandler?
Set maxBytes to 10-100 MB depending on your log volume and disk space. Keep 5-10 backups. For a high-volume service, use TimedRotatingFileHandler (rotate daily) instead.
Can I send logs to multiple remote endpoints?
Yes. Create multiple handler instances (one per endpoint) and attach them all to the logger. Each handler runs independently and failures are isolated.
How do I test a custom handler?
Write a unit test that calls handler.emit() directly with a fake LogRecord, or use Mock to verify that the handler's methods were called with the right arguments.