Python Queue Module: Thread Communication (2026)

The queue module provides thread-safe queues for passing data between threads without explicit locking. Queue.Queue (FIFO), queue.LifoQueue (LIFO), and queue.PriorityQueue handle synchronization internally, allowing clean producer-consumer patterns where one thread produces items and multiple consumer threads process them. Queues are more elegant than raw locks for most multi-threaded designs.

In my early threading projects, I manually coordinated data passing with locks and lists, which led to subtle bugs: items were lost, consumers missed notifications, and deadlocks lurked in corner cases. Switching to queues eliminated those classes of bugs entirely—the queue itself guarantees correctness.

The Basic Queue Pattern

A queue is a thread-safe buffer where threads can safely push items (put()) and pull items (get()). Unlike a list shared between threads, queue operations include built-in synchronization:

import threading
import queue
import time

# Create a thread-safe queue
work_queue = queue.Queue()

def producer():
    """Generate work items and put them on the queue."""
    for i in range(5):
        time.sleep(0.2)
        item = f"task-{i}"
        work_queue.put(item)
        print(f"Produced: {item}")
    # Signal end-of-work with a sentinel value
    work_queue.put(None)

def consumer():
    """Consume work items from the queue."""
    while True:
        item = work_queue.get()
        if item is None:  # Sentinel: no more work
            work_queue.task_done()
            break
        print(f"Consuming: {item}")
        time.sleep(0.1)
        work_queue.task_done()  # Mark this item as processed

# Start one producer and two consumers
prod = threading.Thread(target=producer)
cons1 = threading.Thread(target=consumer)
cons2 = threading.Thread(target=consumer)

prod.start()
cons1.start()
cons2.start()

# Wait for all work to complete
prod.join()
cons1.join()
cons2.join()

print("All work processed")

Key operations:

• put(item) — Add an item to the queue. Blocks if the queue is full (for bounded queues).
• get() — Remove and return an item. Blocks if the queue is empty.
• task_done() — Signal that an item has been processed. Used with join() on the queue.
• queue.join() — Block until all items have been marked with task_done().

The sentinel pattern (putting None to signal end-of-queue) is a common way to tell consumers to stop waiting and exit.

FIFO, LIFO, and Priority Queues

Python provides three queue types for different use cases:

import queue
import threading

# FIFO Queue (first in, first out) — default behavior
fifo = queue.Queue()
fifo.put("task-1")
fifo.put("task-2")
print(fifo.get())  # Output: task-1

# LIFO Queue (last in, first out) — stack-like
lifo = queue.LifoQueue()
lifo.put("task-1")
lifo.put("task-2")
print(lifo.get())  # Output: task-2

# Priority Queue — items are sorted by priority
pq = queue.PriorityQueue()
pq.put((1, "low-priority-task"))
pq.put((0, "high-priority-task"))
priority, task = pq.get()
print(task)  # Output: high-priority-task (priority 0 comes first)

FIFO is ideal for work queues where order matters. LIFO mimics a stack (useful for depth-first processing). PriorityQueue is essential for scheduling where some tasks are more urgent.

Bounded Queues and Backpressure

A bounded queue has a maximum size. When full, put() blocks until a consumer removes an item. This prevents producers from overwhelming the system:

import threading
import queue
import time

# Create a queue with max size 3
bounded_queue = queue.Queue(maxsize=3)

def slow_consumer():
    """Consume slowly, letting the queue fill up and apply backpressure."""
    for i in range(5):
        item = bounded_queue.get()
        print(f"Consuming: {item}")
        time.sleep(2)  # Process slowly
        bounded_queue.task_done()

def aggressive_producer():
    """Try to produce 10 items quickly."""
    for i in range(10):
        print(f"Attempting to produce: task-{i}")
        bounded_queue.put(f"task-{i}")  # Blocks when queue is full
        print(f"Produced: task-{i}")

# The producer will produce task-0, 1, 2, then block until the consumer catches up
producer = threading.Thread(target=aggressive_producer)
consumer = threading.Thread(target=slow_consumer)

producer.start()
consumer.start()

producer.join()
consumer.join()

In this example, the producer creates task-0, 1, 2 and fills the queue. On attempt to produce task-3, put() blocks because the queue is full. Only when the consumer processes task-0 does task-3 get added. This natural backpressure prevents memory exhaustion.

Timeout Patterns

Queue operations accept a timeout parameter to avoid blocking indefinitely:

import queue
import threading
import time

q = queue.Queue()

def try_get_with_timeout():
    """Attempt to get an item, giving up after 2 seconds."""
    try:
        item = q.get(timeout=2)
        print(f"Got: {item}")
    except queue.Empty:
        print("Timeout: no item available after 2 seconds")

# Queue is empty; get() with timeout will wait and then raise Empty
thread = threading.Thread(target=try_get_with_timeout)
thread.start()
thread.join()

# Output: Timeout: no item available after 2 seconds

Timeouts are crucial for avoiding infinite waits, especially when coordinating many threads or when you want a thread to do something else if work isn't available.

Worker Pool with Queue

A common pattern is a fixed pool of worker threads, all consuming from a shared queue:

import queue
import threading
import time

def worker(worker_id, task_queue):
    """A worker thread that processes tasks from the queue."""
    while True:
        try:
            task = task_queue.get(timeout=1)
        except queue.Empty:
            print(f"Worker {worker_id} found no task (timeout)")
            continue
        
        if task is None:  # Sentinel: exit signal
            print(f"Worker {worker_id} exiting")
            task_queue.task_done()
            break
        
        print(f"Worker {worker_id} processing: {task}")
        time.sleep(0.5)
        task_queue.task_done()

# Create a task queue and worker threads
task_queue = queue.Queue()
num_workers = 3

workers = [
    threading.Thread(target=worker, args=(i, task_queue))
    for i in range(num_workers)
]

for w in workers:
    w.start()

# Submit tasks
for i in range(10):
    task_queue.put(f"task-{i}")

# Signal workers to stop
for _ in range(num_workers):
    task_queue.put(None)

# Wait for all tasks to complete
for w in workers:
    w.join()

print("All workers finished")

This pattern scales well: add more workers without changing the task generation code. Each worker independently processes tasks from the shared queue.

Comparison Table: Queue Types

Type	Ordering	Use Case	Get Order
Queue	FIFO	Work distribution, message passing	First in, first out
LifoQueue	LIFO	Depth-first search, backtracking	Last in, first out
PriorityQueue	Priority	Task scheduling, urgent work	Lowest priority value first
SimpleQueue	FIFO	Fast, unbounded FIFO (Python 3.7+)	First in, first out

Key Takeaways

• Queues provide thread-safe data passing without manual locking; use them instead of shared lists.
• FIFO queues are the standard for work distribution; use LIFO or PriorityQueue for specific orderings.
• Bounded queues apply backpressure, preventing producers from overwhelming the system.
• Sentinel values (like None) signal end-of-work to consumers.
• Worker pools—multiple threads consuming from a shared queue—are a powerful pattern for parallel processing.

Frequently Asked Questions

What if a consumer crashes while processing a task?

If the consumer crashes before calling task_done(), the queue will hang forever at queue.join() waiting for that item to be marked complete. Always wrap consumer code in try-finally to ensure task_done() is called.

Is queue.Queue thread-safe without additional locks?

Yes, queue operations (put, get, task_done) are atomic and synchronized internally. However, if you store mutable objects in the queue and modify them without a lock, those modifications can still cause race conditions.

Can I peek at items in a queue without removing them?

No, there's no peek() method in Python's queue module. If you need to examine an item first, pull it out, examine it, and put it back or forward it to another queue.

What's the difference between Queue and SimpleQueue?

Queue supports task_done() and join() for tracking completion; SimpleQueue (Python 3.7+) is a faster, unbounded FIFO without these tracking features. Use Queue for coordinated work; use SimpleQueue for simple message passing.

How do I shutdown a queue gracefully?

Use sentinels (like None) or a flag. Have producers stop adding items and consumers check for stop signals. For urgent shutdown, set a daemon flag on worker threads so the process can exit even if they're blocked on get().

Further Reading

• Python queue module documentation
• Real Python: Queue in Python: A Quick Introduction
• Producer-consumer problem on Wikipedia
• Thread pools using concurrent.futures.ThreadPoolExecutor