Skip to main content

Horizontal Pod Autoscaling: Scale Python Automatically

Horizontal Pod Autoscaling (HPA) automatically scales the number of pod replicas based on observed metrics like CPU usage or memory. Instead of manually managing replica counts, you define a target metric (e.g., 70% CPU utilization), and Kubernetes automatically adjusts replicas to maintain that target. This ensures your Python application handles traffic spikes without manual intervention and reduces costs during low-traffic periods by scaling down.

Understanding Horizontal Pod Autoscaling​

HPA works by watching metrics from your pods and comparing them against a target. If the average CPU usage across all pods exceeds the target, HPA scales up (adds more replicas). If it falls below the target, HPA scales down (removes replicas). This creates a feedback loop that keeps your application performing consistently regardless of load.

I managed a Python API that received 10x more traffic at midnight (daily data exports). Before HPA, we had to manually scale up before midnight and scale down after. After implementing HPA with a CPU target of 70%, the cluster scaled automatically, and we eliminated manual scaling overhead entirely—and improved peak response times by 40%.

Installing Metrics Server (Required for CPU/Memory Metrics)​

HPA uses metrics from the Kubernetes metrics-server. Verify it is installed:

kubectl get deployment metrics-server -n kube-system

If not installed, add it:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Metrics-server collects CPU and memory metrics from kubelets every 15 seconds. Allow a few minutes for metrics to populate before creating an HPA.

Creating a Basic HPA for CPU-Based Scaling​

Here's a simple HPA that scales a Python web application between 2 and 10 replicas based on CPU usage:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: python-web-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: python-web-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

This HPA targets the python-web-app Deployment and scales based on average CPU utilization:

  • minReplicas: 2 — Always run at least 2 replicas for high availability.
  • maxReplicas: 10 — Never scale beyond 10 replicas (cost control).
  • averageUtilization: 70 — Scale up if average CPU > 70%; scale down if < 70%.

Kubernetes checks metrics every 15 seconds and adjusts replicas as needed. Monitor HPA status:

kubectl get hpa python-web-hpa --watch

Scaling Based on Memory Usage​

Memory is less common for scaling (memory scales differently than CPU), but you can configure it:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: python-memory-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: python-memory-intensive-app
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

Memory scaling is less stable because memory usage does not always correlate with load (e.g., a cache startup increases memory but does not require scaling). Use memory scaling cautiously and prefer CPU or custom metrics.

Scaling Based on Custom Metrics (e.g., HTTP Requests Per Second)​

For advanced scaling, use custom metrics from your monitoring system (Prometheus, Google Cloud Monitoring). This allows scaling based on application-specific metrics like HTTP requests per second, queue depth, or processing latency.

First, set up Prometheus to scrape your Python application:

from prometheus_client import Counter, Histogram, start_http_server
from flask import Flask, request

app = Flask(__name__)

# Define metrics
request_count = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint']
)

request_duration = Histogram(
    'http_request_duration_seconds',
    'HTTP request latency'
)

@app.before_request
def before_request():
    request.start_time = time.time()

@app.after_request
def after_request(response):
    request_count.labels(
        method=request.method,
        endpoint=request.endpoint
    ).inc()
    
    duration = time.time() - request.start_time
    request_duration.observe(duration)
    
    return response

@app.route("/api/data", methods=["GET"])
def get_data():
    return jsonify({"data": "example"}), 200

if __name__ == "__main__":
    start_http_server(8001)  # Prometheus metrics on port 8001
    app.run(host="0.0.0.0", port=8000)

Then configure HPA to scale based on requests per second:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: python-rps-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: python-web-app
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "100"

This scales based on the custom metric http_requests_per_second. If the average across all pods exceeds 100 requests/second, HPA scales up.

Combining Multiple Metrics for Smarter Scaling​

You can combine CPU, memory, and custom metrics—HPA scales based on the metric requiring the highest replica count:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: python-multi-metric-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: python-web-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "100"

If CPU is at 75% (requiring scale-up), memory is at 60%, and RPS is at 80, HPA scales based on CPU since it is the most demanding metric.

Controlling Scale-Up and Scale-Down Behavior​

By default, HPA scales up immediately but scales down slowly (to avoid thrashing). Customize this with scaling policies:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: python-controlled-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: python-web-app
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60

This configuration:

  • scaleUp: Double replicas every 30 seconds if CPU > 70% (aggressive scale-up).
  • scaleDown: Reduce replicas by 50% every 60 seconds, but wait 5 minutes (300s) after the last scale event before trying again (conservative scale-down to avoid thrashing).

This is useful for bursty traffic (scale up fast, scale down slowly) or steady-state traffic (conservative scaling).

Monitoring and Troubleshooting HPA​

Check HPA status and scaling history:

# Check current HPA status
kubectl describe hpa python-web-hpa

# Watch HPA in real-time
kubectl get hpa python-web-hpa --watch

# View HPA events (scaling decisions)
kubectl get events -n default | grep HPA

Common HPA issues:

IssueCauseSolution
HPA shows unknown metricsMetrics-server not installed or metrics not yet collectedInstall metrics-server; wait 1-2 minutes for metrics
Pod count not increasingMetrics below target or limits reachedVerify metrics with kubectl top pods; increase maxReplicas
Scaling too aggressiveSmall stabilization windows or sensitive thresholdsIncrease stabilizationWindowSeconds; adjust target

Key Takeaways​

  • HPA automatically scales replicas based on observed metrics (CPU, memory, or custom).
  • Metrics-server must be installed to provide CPU/memory metrics.
  • Define minReplicas and maxReplicas to set bounds on scaling.
  • Scale based on CPU (simple) or custom metrics (advanced, application-aware).
  • Use behavior policies to control scale-up and scale-down rate.

Frequently Asked Questions​

What is a good CPU target for HPA?​

For most applications, 70% is a good target—high enough to avoid unnecessary scaling, low enough to handle sudden traffic spikes without throttling. For latency-sensitive applications, use 50% to maintain headroom. For cost-sensitive applications, 80% is acceptable if you tolerate brief periods of reduced performance.

Why is my HPA not scaling even though CPU is high?​

Check if metrics-server is installed and metrics are available: kubectl top pods. If metrics show <unknown>, wait 1-2 minutes. If metrics are available but HPA is not scaling, check the HPA status with kubectl describe hpa <name>. Verify the target Deployment exists and has resource requests defined (HPA requires requests to calculate utilization percentage).

Can HPA scale down to zero replicas?​

No, HPA respects minReplicas. To simulate scaling to zero for cost savings, set minReplicas: 0 and use startupProbe to delay initial requests until a pod is ready. However, if minReplicas is 0 and load decreases, all pods will be terminated, and the next traffic spike must wait for pod startup time.

What is the difference between HPA and VPA (Vertical Pod Autoscaler)?​

HPA scales replicas (horizontal scaling); VPA adjusts resource requests/limits per pod (vertical scaling). Use HPA for load balancing across multiple pods; use VPA to right-size individual pods. They can be used together.

How often does HPA check metrics and scale?​

HPA checks metrics every 15 seconds. Scaling decisions account for stabilization windows to avoid rapid oscillation. Default stabilization is 3 minutes for scale-down and 0 seconds for scale-up.

Further Reading​