Configuration Layers and Precedence: Multi-Environment

As your Python application grows from a local script to a deployed service, you need configuration to adapt: development might use SQLite and verbose logging, staging might use PostgreSQL and normal logging, and production might use managed cloud databases and minimal logging. Configuration layering solves this by stacking multiple configuration sources with clear precedence rules. Each layer is a configuration source (hard defaults, files, environment variables, secret managers) and they are merged in order, with higher-priority layers overriding lower ones.

The key principle: configuration flows from broadest (application defaults) to most specific (environment variables). This prevents environment-bleeding bugs where a developer's .env setting accidentally affects production, or where a production secret leaks into local development. This article covers the mental model of layered configuration, practical implementation patterns, and how to debug precedence issues.

The Configuration Hierarchy: From Defaults to Overrides

Think of configuration as a series of layers, each potentially overriding the layer below:

Layer 5 (Highest Priority): Environment Variables
    |
Layer 4: Secret Manager (AWS Secrets Manager, Vault)
    |
Layer 3: Environment-Specific Files (.env.production, config.staging.yaml)
    |
Layer 2: Shared Configuration Files (.env, config.yaml)
    |
Layer 1 (Lowest Priority): Application Defaults (hardcoded or config module)

A concrete example:

import os
from dotenv import load_dotenv

# Layer 1: Hard defaults in code
config = {
    "DEBUG": False,
    "DATABASE_HOST": "localhost",
    "DATABASE_PORT": 5432,
    "LOG_LEVEL": "INFO",
    "MAX_CONNECTIONS": 10,
}

# Layer 2: Load shared .env file
load_dotenv(".env")

# Layer 3: Load environment-specific overrides
env = os.getenv("ENVIRONMENT", "development")
load_dotenv(f".env.{env}", override=True)

# Layer 4: Direct environment variables override everything
for key in ["DEBUG", "DATABASE_HOST", "LOG_LEVEL"]:
    env_val = os.getenv(key)
    if env_val is not None:
        config[key] = env_val

# Resulting config respects precedence
print(f"DEBUG mode: {config['DEBUG']}")
print(f"Database: {config['DATABASE_HOST']}:{config['DATABASE_PORT']}")

This ensures that an environment variable set in your production deployment overrides any .env file or code default, preventing accidental use of development credentials.

Practical Multi-Environment Setup

A typical project structure:

my-app/
├── config.py              # Layer 1: hard defaults + merge logic
├── .env                   # Layer 2: shared development defaults (Git-ignored)
├── .env.example           # Committed template
├── .env.development       # Layer 3: development overrides (optional)
├── .env.staging           # Layer 3: staging overrides
├── config.staging.yaml    # Optional: richer config for staging
└── src/
    └── app.py

In config.py, define the complete merge logic:

import os
from pathlib import Path
from dotenv import load_dotenv

class Config:
    """Base configuration with sensible defaults."""
    DEBUG = False
    DATABASE_HOST = "localhost"
    DATABASE_PORT = 5432
    DATABASE_NAME = "myapp"
    CACHE_TTL = 3600
    LOG_LEVEL = "INFO"
    API_TIMEOUT = 30

class DevelopmentConfig(Config):
    """Override for local development."""
    DEBUG = True
    CACHE_TTL = 60
    LOG_LEVEL = "DEBUG"

class StagingConfig(Config):
    """Override for staging environment."""
    DATABASE_HOST = "staging-db.internal"
    LOG_LEVEL = "INFO"

class ProductionConfig(Config):
    """Override for production (mostly secrets from env vars)."""
    DEBUG = False
    LOG_LEVEL = "WARNING"

# Select config class based on environment
def get_config():
    env = os.getenv("ENVIRONMENT", "development").lower()
    
    # Layer 1: Class defaults
    if env == "production":
        config = ProductionConfig()
    elif env == "staging":
        config = StagingConfig()
    else:
        config = DevelopmentConfig()
    
    # Layer 2: Load .env file
    load_dotenv(override=False)
    
    # Layer 3: Load environment-specific .env
    load_dotenv(f".env.{env}", override=True)
    
    # Layer 4: Apply environment variables to config object
    for key in dir(config):
        if key.isupper():
            env_val = os.getenv(key)
            if env_val is not None:
                # Type coercion for booleans and integers
                if isinstance(getattr(config, key), bool):
                    setattr(config, key, env_val.lower() in ("true", "1", "yes"))
                elif isinstance(getattr(config, key), int):
                    try:
                        setattr(config, key, int(env_val))
                    except ValueError:
                        pass
                else:
                    setattr(config, key, env_val)
    
    return config

# At application startup
if __name__ == "__main__":
    cfg = get_config()
    print(f"Running in {os.getenv('ENVIRONMENT', 'development')} mode")
    print(f"Database: {cfg.DATABASE_HOST}:{cfg.DATABASE_PORT}")
    print(f"Debug: {cfg.DEBUG}")

In your application:

from config import get_config

cfg = get_config()

# Use cfg throughout
if cfg.DEBUG:
    print("Debug mode enabled")

db_conn = connect(cfg.DATABASE_HOST, cfg.DATABASE_PORT)

Precedence Rules and Debugging

To avoid confusion, document your precedence order explicitly:

"""
Configuration Precedence (highest to lowest):
1. Environment variables (set at deploy time)
2. .env.{ENVIRONMENT} file (e.g., .env.production)
3. .env file (shared defaults, Git-ignored)
4. Config class defaults (Config, DevelopmentConfig, etc.)

Example: DATABASE_HOST
- Hardcoded default in Config: "localhost"
- .env shared: "dev-db.example.com"
- .env.production: "prod-db.internal"
- Environment variable: "prod-db.internal.region-2"
→ Final value: "prod-db.internal.region-2"
"""

Debug precedence issues by printing the config flow:

import os

def load_config_with_debug():
    config = {}
    
    # Layer 1
    config.update({"DATABASE_HOST": "localhost"})
    print(f"[Layer 1] Database: {config['DATABASE_HOST']}")
    
    # Layer 2
    from dotenv import dotenv_values
    env_values = dotenv_values(".env")
    config.update(env_values)
    print(f"[Layer 2] Database: {config.get('DATABASE_HOST', 'N/A')}")
    
    # Layer 3
    prod_values = dotenv_values(".env.production")
    if prod_values:
        config.update(prod_values)
        print(f"[Layer 3] Database: {config.get('DATABASE_HOST', 'N/A')}")
    
    # Layer 4
    env_var = os.getenv("DATABASE_HOST")
    if env_var:
        config["DATABASE_HOST"] = env_var
        print(f"[Layer 4] Database: {config['DATABASE_HOST']} (from env var)")
    
    return config

cfg = load_config_with_debug()

Avoiding Common Configuration Mistakes

A comparison of risky vs safe patterns:

Pattern	Risk	Safe Alternative
Hardcode secrets in code	Leaks in version control	Layer them from environment vars
Load production .env into dev	Environment bleeding, accidental data access	Use separate .env.production ignored in Git
Skip environment validation	Silent failures, wrong database used	Validate and log which config layer is active
Override layer order unpredictably	Mystery bugs where changes do not apply	Document precedence clearly, test each layer

Key Takeaways

• Configuration layering allows the same codebase to adapt to development, staging, and production by stacking configuration sources with clear precedence.
• Always define hard defaults in code (Layer 1), then progressively override with files and environment variables (Layers 2–4).
• Use separate .env.{environment} files for environment-specific overrides and keep them Git-ignored in production environments.
• Document your precedence order explicitly and provide debug logging to trace which layer provided each setting.
• Test each configuration layer independently to catch environment-bleeding bugs early.

Frequently Asked Questions

How do I test that my configuration layers work correctly?

Write unit tests that set environment variables and load each layer. Use pytest fixtures to isolate each test: @pytest.fixture(autouse=True) def reset_env(monkeypatch): monkeypatch.delenv("DATABASE_HOST", raising=False). Assert that each layer correctly overrides the previous one.

Can I use YAML or JSON files in configuration layers?

Yes. Use libraries like yaml, json, or toml to load files, then merge them into your config dict: import yaml; config.update(yaml.safe_load(open(".env.staging.yaml"))). Place YAML loading after environment variables so env vars still override file values.

What if I need the same secret (e.g., API key) in multiple environments but with different values?

Use environment-specific .env files with different values: .env.staging has API_KEY=staging_key_123, .env.production has API_KEY=prod_key_456. At deploy time, the orchestrator (kubectl, Docker Compose, CI/CD) injects the correct .env.{environment} file. Alternatively, store each environment's secrets in a secret manager and let the app fetch them.

How do I rotate secrets without redeploying?

Use a secret manager (AWS Secrets Manager, Vault) as Layer 4, before environment variables: the app periodically fetches fresh secrets without a code change. For quick rotation without secret manager: trigger a configuration reload from a /reload-config endpoint (careful: check authorization), or use a sidecar process that updates a config file watched by your app.

Should I commit .env.example with real (dummy) values or blanks?

Include dummy or minimal example values so developers know the format and what to expect: API_KEY=sk_test_example or API_KEY=your_key_here. Never include real development or production secrets in .env.example. This allows new developers to copy and understand what to fill in.

Further Reading

• Twelve-Factor App: Store config in environment
• Python Configuration Management Patterns
• Pydantic Settings for advanced config
• Managing secrets in Kubernetes