Skip to main content

Metrics in Python: Collect and Export Performance Data

Metrics are numeric measurements of your application's behavior: request counts, response times, error rates, memory usage, database pool size. While logs tell you what happened, metrics tell you how often and how fast. A time-series database like Prometheus stores metrics with timestamps, allowing you to build dashboards that show trends, set alerts on thresholds, and correlate performance degradation across services. Logs are for debugging specific incidents; metrics are for operational visibility.

Instrumentation—the practice of adding metric collection to your code—is straightforward in Python. You use libraries like prometheus_client (the standard) or OpenTelemetry (vendor-neutral) to define and update metrics. The library exports metrics in a standard format (Prometheus text format, OTLP) that systems like Prometheus, Grafana, Datadog, and CloudWatch can ingest. This separation of concern means you never hard-code metric names or endpoints into your application; the infrastructure is responsible for routing metrics to the correct backend.

What Are the Four Metric Types?​

Prometheus and most observability systems define four fundamental metric types:

Metric TypeExampleWhen to Use
CounterTotal API requests served: 4,257,892Count of events that only increase (never decrease)
GaugeCurrent memory usage: 512 MBPoint-in-time measurements that can go up or down
HistogramResponse time distribution: p50=45ms, p95=230ms, p99=1200msMeasure value distributions and percentiles
SummarySame as histogram but computed differentlyAlternative to histogram (older, less efficient)

A counter starts at 0 and only increases (e.g., total requests). A gauge fluctuates (e.g., memory used). A histogram records multiple measurements and lets you query percentiles (how long were the slowest 1% of requests?). Most applications rely on counters and histograms for request-level metrics, and gauges for resource utilization.

How Do You Instrument a Python Application with prometheus_client?​

The prometheus_client library is the Python Prometheus client. You define metrics at module level, then update them in your code:

from prometheus_client import Counter, Histogram, Gauge
import time

# Define metrics
request_count = Counter('requests_total', 'Total HTTP requests', ['method', 'endpoint'])
request_duration = Histogram('request_duration_seconds', 'HTTP request latency', ['endpoint'])
active_connections = Gauge('active_connections', 'Number of active database connections')

def handle_request(method, endpoint):
    """Simulated request handler."""
    request_count.labels(method=method, endpoint=endpoint).inc()
    
    with request_duration.labels(endpoint=endpoint).time():
        # Measure execution time automatically
        active_connections.set(get_db_connection_count())
        result = do_work()
        time.sleep(0.1)  # Simulate work
    
    return result

def get_db_connection_count():
    """Return current connection pool size."""
    return 5  # Placeholder

# Simulate requests
handle_request('GET', '/users')
handle_request('POST', '/orders')
handle_request('GET', '/users')

Each metric is identified by a name (e.g., requests_total) and optional labels (dimensions like method and endpoint). When you export metrics, they appear as:

# HELP requests_total Total HTTP requests
# TYPE requests_total counter
requests_total{method="GET",endpoint="/users"} 2
requests_total{method="POST",endpoint="/orders"} 1
request_duration_seconds_bucket{endpoint="/users",le="0.005"} 0
request_duration_seconds_bucket{endpoint="/users",le="0.01"} 1
request_duration_seconds_bucket{endpoint="/users",le="+Inf"} 2
request_duration_seconds_sum{endpoint="/users"} 0.203
request_duration_seconds_count{endpoint="/users"} 2
active_connections 5

The Prometheus scraper (on the ops side) polls your application periodically and collects these metrics.

How Do You Export Metrics to Prometheus?​

Prometheus expects metrics to be available at an HTTP endpoint, typically /metrics. The prometheus_client library provides a Flask/Django integration:

from flask import Flask
from prometheus_client import Counter, generate_latest, CONTENT_TYPE_LATEST

app = Flask(__name__)

request_count = Counter('flask_requests_total', 'Total Flask requests', ['method', 'endpoint'])

@app.before_request
def before_request():
    """Hook called before every request."""
    pass

@app.after_request
def after_request(response):
    """Hook called after every request."""
    from flask import request
    request_count.labels(
        method=request.method,
        endpoint=request.endpoint or 'unknown'
    ).inc()
    return response

@app.route('/metrics')
def metrics():
    """Expose metrics in Prometheus format."""
    return generate_latest(), 200, {'Content-Type': CONTENT_TYPE_LATEST}

@app.route('/users')
def get_users():
    return {'users': []}

if __name__ == '__main__':
    app.run()

Prometheus configuration (prometheus.yml) then scrapes your app:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'python-app'
    static_configs:
      - targets: ['localhost:5000']
    metrics_path: '/metrics'

Every 15 seconds, Prometheus POSTs to http://localhost:5000/metrics and stores the returned metrics in its time-series database.

How Do You Instrument Specific Functions with Decorators?​

For common patterns (tracking function execution time, counting calls), decorators eliminate boilerplate:

import functools
import time
from prometheus_client import Counter, Histogram

# Define metrics
function_calls = Counter('function_calls_total', 'Total function calls', ['function'])
function_duration = Histogram('function_duration_seconds', 'Function execution time', ['function'])

def track_metrics(func):
    """Decorator to track function calls and duration."""
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        function_calls.labels(function=func.__name__).inc()
        start = time.time()
        try:
            return func(*args, **kwargs)
        finally:
            duration = time.time() - start
            function_duration.labels(function=func.__name__).observe(duration)
    return wrapper

@track_metrics
def slow_operation():
    time.sleep(0.5)
    return 'done'

slow_operation()  # Metrics recorded automatically

This pattern is much cleaner than manually calling metric.inc() in every function.

What Metrics Should You Instrument in a Typical Application?​

Start with the golden signals: latency, traffic, errors, and saturation.

  • Latency: request duration, database query time, external API call duration
  • Traffic: requests per second, queries per second, messages processed per second
  • Errors: 4xx responses, 5xx responses, exception counts by type
  • Saturation: database connection pool utilization, memory usage, CPU usage
from prometheus_client import Counter, Histogram, Gauge
import time

# Golden signals
http_requests = Counter('http_requests_total', 'Total HTTP requests', ['method', 'status'])
http_duration = Histogram('http_request_duration_seconds', 'HTTP request latency', ['method'])
http_errors = Counter('http_errors_total', 'Total HTTP errors', ['method', 'error_type'])

db_pool_usage = Gauge('db_pool_connections_used', 'Database connections in use')
db_query_duration = Histogram('db_query_duration_seconds', 'Database query latency')

def handle_http_request(method, path):
    http_duration.labels(method=method).time()  # Context manager
    try:
        status = do_request(method, path)
        http_requests.labels(method=method, status=status).inc()
    except Exception as e:
        http_errors.labels(method=method, error_type=type(e).__name__).inc()
        raise

def query_database(sql):
    start = time.time()
    try:
        db_pool_usage.inc()
        return execute(sql)
    finally:
        duration = time.time() - start
        db_query_duration.observe(duration)
        db_pool_usage.dec()

These metrics give operators the visibility to understand both normal operation and degradation.

Key Takeaways​

  • Four metric types: counters (total), gauges (current value), histograms (distributions), summaries (old histograms).
  • Define metrics at module level, update them in your code.
  • Use prometheus_client to instrument Python applications.
  • Export metrics via a /metrics HTTP endpoint.
  • Instrument golden signals: latency, traffic, errors, saturation.

Frequently Asked Questions​

Should I instrument every function or just critical paths?​

Instrument critical paths and user-facing operations first. Adding instrumentation is cheap (a counter increment is microseconds), but collecting and storing millions of metrics can be expensive. Start broad, then refine.

What is the difference between a Counter and a Gauge?​

A Counter only increases (total requests ever served). A Gauge can go up or down (memory used now). Use Counter for totals; use Gauge for current state.

How do I track metric values that change infrequently?​

Use a Gauge and set its value periodically (e.g., in a background thread every 60 seconds). Or emit the value when it changes.

Can I send the same metric to multiple backends?​

Yes. The prometheus_client library exports metrics in a standard format. You can scrape /metrics from Prometheus, CloudWatch, Datadog, New Relic, etc. simultaneously.

How do I avoid high cardinality metrics that explode storage?​

Be careful with label values. Never use user_id or request_id as a label (infinite cardinality). Use status codes, endpoints, methods, error types—values with bounded cardinality.

Further Reading​