Nested reentrant context managers explained
Context managers can be nested (multiple context managers in a single with statement) or reentrant (the same context manager entered multiple times concurrently). Nested managers are the common case: acquiring multiple resources in a predictable order and releasing them in reverse order. Reentrant managers are trickier: they must be thread-safe or async-safe and track state per entry. Understanding both patterns is essential for writing robust, concurrent Python code.
Nesting is straightforward: Python enters left-to-right and exits right-to-left (stack discipline). Reentrancy is harder: you must use thread-local storage or async context to track per-entry state, or implement reference counting. Real-world examples include connection pools, thread locks, temporary state changes, and resource acquisition across async tasks.
Nested Context Managers: Basics and Order
When you nest context managers with a comma, Python enters each in left-to-right order and exits in right-to-left order (LIFO). This order is crucial: resources acquired later often depend on earlier ones, and must be released first.
# Example 1: Simple nesting with clear order
class Resource:
def __init__(self, name):
self.name = name
def __enter__(self):
print(f"{self.name}: acquiring")
return self
def __exit__(self, exc_type, exc_val, exc_tb):
print(f"{self.name}: releasing")
return False
# Comma syntax: acquire A, B, C in order; release C, B, A
print("--- Nested resources (comma syntax) ---")
with Resource("A") as a, \
Resource("B") as b, \
Resource("C") as c:
print(f"Using {a.name}, {b.name}, {c.name}")
print()
# Equivalent with nested with statements
print("--- Nested resources (nested with) ---")
with Resource("A") as a:
with Resource("B") as b:
with Resource("C") as c:
print(f"Using {a.name}, {b.name}, {c.name}")
# Output:
# --- Nested resources (comma syntax) ---
# A: acquiring
# B: acquiring
# C: acquiring
# Using A, B, C
# C: releasing
# B: releasing
# A: releasing
#
# --- Nested resources (nested with) ---
# A: acquiring
# B: acquiring
# C: acquiring
# Using A, B, C
# C: releasing
# B: releasing
# A: releasing
Example: Database Transaction with Connection Pool
A realistic scenario: acquire a connection from a pool, start a transaction, and ensure both are released in the correct order.
from contextlib import contextmanager
class ConnectionPool:
def __init__(self, size=3):
self.connections = [f"conn_{i}" for i in range(size)]
self.available = list(self.connections)
@contextmanager
def get_connection(self):
"""Acquire a connection from the pool, release when done."""
if not self.available:
raise RuntimeError("No connections available")
conn = self.available.pop()
print(f"Acquired: {conn}")
try:
yield conn
finally:
print(f"Released: {conn}")
self.available.append(conn)
class Transaction:
def __init__(self, connection):
self.connection = connection
self.id = None
@contextmanager
def __call__(self):
"""Start a transaction, commit or rollback on exit."""
import time
self.id = f"txn_{int(time.time() * 1000) % 10000}"
print(f"Starting {self.id} on {self.connection}")
try:
yield self
print(f"Committing {self.id}")
except Exception as e:
print(f"Rolling back {self.id} due to {type(e).__name__}")
raise
pool = ConnectionPool()
txn_factory = Transaction
print("--- Nested: connection then transaction ---")
with pool.get_connection() as conn:
txn = txn_factory(conn)
with txn():
print(f"Executing query on {conn}")
print("\n--- Multiple transactions on one connection ---")
with pool.get_connection() as conn:
txn = txn_factory(conn)
with txn():
print("Transaction 1")
with txn():
print("Transaction 2")
# Output:
# --- Nested: connection then transaction ---
# Acquired: conn_2
# Starting txn_1234 on conn_2
# Executing query on conn_2
# Committing txn_1234
# Released: conn_2
#
# --- Multiple transactions on one connection ---
# Acquired: conn_1
# Starting txn_5678 on conn_1
# Transaction 1
# Committing txn_5678
# Starting txn_5679 on conn_1
# Transaction 2
# Committing txn_5679
# Released: conn_1
Reentrant Context Managers: The Challenge
A reentrant context manager is one that can be entered multiple times, potentially concurrently (in threads) or sequentially in async contexts. The challenge is tracking state separately for each entry. Without careful implementation, nested entries will interfere: the first exit will clean up resources still needed by the second entry.
# Example 2: Non-reentrant (broken) context manager
import threading
class NonReentrant:
def __init__(self, name):
self.name = name
self.state = None
def __enter__(self):
print(f"{self.name} [{threading.current_thread().name}]: acquiring")
self.state = "open"
return self
def __exit__(self, exc_type, exc_val, exc_tb):
print(f"{self.name} [{threading.current_thread().name}]: releasing (state was {self.state})")
self.state = "closed"
return False
cm = NonReentrant("shared")
print("--- Sequential reentrancy (broken) ---")
with cm:
print(f"Entry 1: state = {cm.state}")
with cm: # Second entry overwrites state
print(f"Entry 2: state = {cm.state}")
print(f"Back to entry 1: state = {cm.state}") # Should be "open" but is "closed"!
# Output:
# --- Sequential reentrancy (broken) ---
# shared [MainThread]: acquiring
# Entry 1: state = open
# shared [MainThread]: releasing (state was open)
# Entry 2: state = closed
# Back to entry 1: state = closed
Reentrant Solution 1: Reference Counting
Reference counting tracks how many times the context manager has been entered and only cleans up on the final exit.
# Example 3: Reentrant with reference counting
import threading
class ReentrantWithRefCount:
def __init__(self, name):
self.name = name
self.ref_count = 0
self.lock = threading.Lock()
def __enter__(self):
with self.lock:
self.ref_count += 1
print(f"{self.name}: acquiring (ref_count = {self.ref_count})")
return self
def __exit__(self, exc_type, exc_val, exc_tb):
with self.lock:
self.ref_count -= 1
if self.ref_count == 0:
print(f"{self.name}: releasing (ref_count = 0)")
else:
print(f"{self.name}: not releasing (ref_count = {self.ref_count})")
return False
cm = ReentrantWithRefCount("shared")
print("--- Sequential reentrancy with ref counting ---")
with cm:
print(f"Entry 1: ref_count = {cm.ref_count}")
with cm:
print(f"Entry 2: ref_count = {cm.ref_count}")
print(f"Back to entry 1: ref_count = {cm.ref_count}")
print(f"All done: ref_count = {cm.ref_count}")
# Output:
# --- Sequential reentrancy with ref counting ---
# shared: acquiring (ref_count = 1)
# Entry 1: ref_count = 1
# shared: acquiring (ref_count = 2)
# Entry 2: ref_count = 2
# shared: not releasing (ref_count = 1)
# Back to entry 1: ref_count = 1
# shared: releasing (ref_count = 0)
# All done: ref_count = 0
Reentrant Solution 2: Thread-Local Storage
For thread-safe reentrancy, use threading.local() to store state per thread.
# Example 4: Thread-local state for reentrancy
import threading
from contextlib import contextmanager
class ThreadLocalContext:
def __init__(self, name):
self.name = name
self.local = threading.local()
@contextmanager
def __call__(self):
"""Return a context manager for this thread."""
if not hasattr(self.local, 'stack'):
self.local.stack = []
self.local.stack.append(self.name)
thread_name = threading.current_thread().name
print(f"{self.name} [{thread_name}]: entering (stack = {self.local.stack})")
try:
yield self
finally:
self.local.stack.pop()
print(f"{self.name} [{thread_name}]: exiting (stack = {self.local.stack})")
ctx = ThreadLocalContext("resource")
def worker(worker_id):
for i in range(2):
with ctx():
print(f"Worker {worker_id} iteration {i}")
print("--- Thread-local reentrancy ---")
threads = [
threading.Thread(target=worker, args=(i,), name=f"Worker-{i}")
for i in range(2)
]
for t in threads:
t.start()
for t in threads:
t.join()
# Output:
# --- Thread-local reentrancy ---
# resource [Worker-0]: entering (stack = ['resource'])
# Worker 0 iteration 0
# resource [Worker-0]: exiting (stack = [])
# resource [Worker-0]: entering (stack = ['resource'])
# Worker 0 iteration 1
# resource [Worker-0]: exiting (stack = [])
# resource [Worker-1]: entering (stack = ['resource'])
# Worker 1 iteration 0
# resource [Worker-1]: exiting (stack = [])
# resource [Worker-1]: entering (stack = ['resource'])
# Worker 1 iteration 1
# resource [Worker-1]: exiting (stack = [])
Reentrant Solution 3: Stack-Based Per-Entry State
Store state in a stack, with one entry per with block. This works for both sequential and nested reentrancy.
# Example 5: Stack-based state for complex reentrancy
class ReentrantWithStack:
def __init__(self, name):
self.name = name
self.state_stack = []
def __enter__(self):
state = {
"level": len(self.state_stack),
"data": f"state_{len(self.state_stack)}"
}
self.state_stack.append(state)
print(f"{self.name}: acquiring level {state['level']}")
return state
def __exit__(self, exc_type, exc_val, exc_tb):
if not self.state_stack:
raise RuntimeError("Stack underflow")
state = self.state_stack.pop()
print(f"{self.name}: releasing level {state['level']}")
return False
cm = ReentrantWithStack("resource")
print("--- Stack-based reentrancy ---")
with cm as s1:
print(f"Entry 1: level = {s1['level']}, data = {s1['data']}")
with cm as s2:
print(f"Entry 2: level = {s2['level']}, data = {s2['data']}")
with cm as s3:
print(f"Entry 3: level = {s3['level']}, data = {s3['data']}")
print(f"Back to entry 1: stack size = {len(cm.state_stack)}")
# Output:
# --- Stack-based reentrancy ---
# resource: acquiring level 0
# Entry 1: level = 0, data = state_0
# resource: acquiring level 1
# Entry 2: level = 1, data = state_1
# resource: acquiring level 2
# Entry 3: level = 2, data = state_2
# resource: releasing level 2
# resource: releasing level 1
# Back to entry 1: stack size = 1
# resource: releasing level 0
When to Use Nesting vs. Reentrancy
| Pattern | Use Case | Example |
|---|---|---|
| Nested | Multiple independent resources | Multiple file handles, DB connection + transaction |
| Reentrant | Recursive algorithms | Recursive function with resource guards |
| Reentrant | Thread pools | Thread-safe resource access from multiple threads |
| Reentrant | Async contexts | Multiple async tasks sharing a resource |
Key Takeaways
- Nested context managers acquire left-to-right and release right-to-left (LIFO); this order is critical for correctness.
- Reentrant context managers can be entered multiple times; they require careful state management via reference counting, thread-local storage, or stacks.
- Use comma syntax (
with A() as a, B() as b:) for readable nesting of 2-3 resources; use nestedwithfor complex dependencies. - For thread-safe reentrancy, use
threading.local()to isolate state per thread. - Stack-based state works for both sequential and deeply nested reentrancy.
Frequently Asked Questions
Why is exit order right-to-left?
Resource B may depend on resource A. If A is released before B, using B after that point will fail. LIFO order ensures dependent resources are released in the correct order.
Can I use the same context manager instance in multiple threads?
Yes, if it is reentrant. Use threading.Lock() to protect shared state and threading.local() to isolate per-thread state. Non-reentrant context managers in multiple threads will corrupt state.
What if nesting depth is unknown at write time?
Use contextlib.ExitStack to dynamically manage cleanup functions. Register cleanup in a loop, and ExitStack calls them in LIFO order.
How do I make an async-reentrant context manager?
Use asyncio.Lock() instead of threading.Lock() and avoid blocking operations. Store per-task state using contextvars.ContextVar for isolation across async tasks.
Is reference counting thread-safe?
Only if you protect it with a lock, as shown in Example 3. Unprotected increment/decrement can race and cause state corruption.