Skip to main content

gRPC Code Generation: Python Stub Tutorial

The grpc_tools.protoc compiler transforms .proto files into Python classes and gRPC stubs—a critical build step in any gRPC project. Understanding the generated code structure, build options, and CI/CD integration prevents runtime import errors and ensures reproducibility across your team. The Python code generator outputs two files per proto: a message definitions file (*_pb2.py, 600–2000 lines) and a gRPC services file (*_pb2_grpc.py, 300–800 lines). This guide covers the full compilation workflow, customizing output, and debugging common generation issues.

The Compilation Pipeline​

The protoc compiler (bundled with grpc-tools) reads .proto files, validates syntax, and generates language-specific code:

# Install the code-generation tools
pip install grpcio-tools

# Compile a single proto file
python -m grpc_tools.protoc \
  -I protos \
  --python_out=generated \
  --grpc_python_out=generated \
  protos/service.proto

Flags explained:

  • -I protos: Include path for imports (proto dependencies)
  • --python_out=generated: Output directory for *_pb2.py message definitions
  • --grpc_python_out=generated: Output directory for *_pb2_grpc.py service stubs

This produces:

  • generated/service_pb2.py: Message classes (Order, OrderResponse, etc.)
  • generated/service_pb2_grpc.py: Servicer and Stub classes

Batch Compilation for Multi-File Projects​

For larger projects with nested proto directories, compile all files at once:

# Compile all protos recursively
python -m grpc_tools.protoc \
  -I protos \
  --python_out=generated \
  --grpc_python_out=generated \
  $(find protos -name "*.proto")

# Or use a makefile for repeatability
# Makefile
.PHONY: generate-stubs
generate-stubs:
	find protos -name "*.proto" | \
	  xargs python -m grpc_tools.protoc \
	    -I protos \
	    --python_out=generated \
	    --grpc_python_out=generated

.PHONY: clean-stubs
clean-stubs:
	find generated -name "*_pb2*.py" -delete

Run make generate-stubs whenever you modify a .proto file. This keeps your build repeatable and automated.

Understanding Generated Files​

After compiling orders.proto, inspect the generated code:

# orders_pb2.py (excerpt)
# Message class for Order
class Order(message.Message):
    order_id: str
    quantity: int
    price: float
    def __init__(self, order_id: str = "", quantity: int = 0, price: float = 0.0) -> None: ...
    def SerializeToString(self) -> bytes: ...
    @staticmethod
    def FromString(data: bytes) -> "Order": ...

# Descriptor metadata (used internally by gRPC runtime)
DESCRIPTOR = _descriptor.FileDescriptor(...)
# orders_pb2_grpc.py (excerpt)
# Servicer base class (implement this on the server)
class OrderServiceServicer(object):
    def CreateOrder(self, request, context):
        raise NotImplementedError()
    
    def GetOrder(self, request, context):
        raise NotImplementedError()

# Stub class (use this on the client)
class OrderServiceStub(object):
    def __init__(self, channel):
        self.CreateOrder = channel.unary_unary(
            "/ecommerce.orders.OrderService/CreateOrder",
            request_serializer=Order.SerializeToString,
            response_deserializer=OrderResponse.FromString,
        )

The .Servicer class is a template for server implementations; the .Stub class provides client methods. The DESCRIPTOR metadata is used by gRPC internally to route calls and serialize messages.

Integration with Python Imports​

Generated stubs must be importable. Set up your project structure:

project/
├── protos/
│   └── orders.proto
├── generated/
│   ├── __init__.py
│   ├── orders_pb2.py
│   └── orders_pb2_grpc.py
├── services/
│   ├── __init__.py
│   └── order_service.py
└── Makefile

Create generated/__init__.py (can be empty):

# generated/__init__.py
# (empty, just makes the directory a Python package)

In your service code, import the generated stubs:

# services/order_service.py
import sys
sys.path.insert(0, "../generated")

from orders_pb2 import Order, OrderResponse
from orders_pb2_grpc import OrderServiceServicer, add_OrderServiceServicer_to_server

Or configure your Python path at runtime:

import os
import sys

# Ensure generated stubs are importable
GENERATED_PATH = os.path.join(os.path.dirname(__file__), "../generated")
if GENERATED_PATH not in sys.path:
    sys.path.insert(0, GENERATED_PATH)

Handling Proto Imports and Namespaces​

When your protos import other protos, the import chain must be resolvable:

// protos/common/types.proto
syntax = "proto3";
package common;

message Money {
  int64 amount_cents = 1;
}
// protos/orders/service.proto
syntax = "proto3";
package ecommerce.orders;

import "common/types.proto";

message Order {
  common.Money total = 1;
}

Compile with the root proto directory as the include path:

python -m grpc_tools.protoc \
  -I protos \
  --python_out=generated \
  --grpc_python_out=generated \
  protos/common/types.proto \
  protos/orders/service.proto

Generated imports will be relative:

# generated/orders_pb2.py (excerpt)
import common.types_pb2 as common_dot_types__pb2
# Expects: generated/common/types_pb2.py

Ensure your output structure mirrors the proto directory layout to avoid import errors.

Custom Plugin Options​

The protoc compiler supports plugin options (e.g., --python_out=OPTION:DIR):

# Enable pyi (type hints) generation for better IDE support
python -m grpc_tools.protoc \
  -I protos \
  --python_out=pyi_out:generated \
  --grpc_python_out=grpc_asyncio:generated \
  protos/orders.proto

Options:

  • pyi_out: Generate .pyi stub files for type checking (Python 3.8+, improves IDE autocomplete)
  • grpc_asyncio: Generate async/await stubs (instead of synchronous stubs)

For async gRPC (recommended for high-concurrency services):

python -m grpc_tools.protoc \
  -I protos \
  --python_out=generated \
  --grpc_python_out=grpc_asyncio:generated \
  protos/orders.proto

This generates async servicers and stubs suitable for asyncio:

# Async server
class OrderServiceServicer:
    async def CreateOrder(self, request, context):
        return OrderResponse(status="created")

# Async client
async with grpc.aio.insecure_channel("localhost:50051") as channel:
    stub = OrderServiceStub(channel)
    response = await stub.CreateOrder(order)

CI/CD Integration: Automated Code Generation​

Add proto compilation to your CI/CD pipeline (GitHub Actions example):

# .github/workflows/generate-stubs.yml
name: Generate gRPC Stubs

on: [push, pull_request]

jobs:
  generate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.11"
      
      - name: Install grpc-tools
        run: pip install grpcio-tools
      
      - name: Compile protos
        run: |
          find protos -name "*.proto" | xargs \
            python -m grpc_tools.protoc \
              -I protos \
              --python_out=generated \
              --grpc_python_out=grpc_asyncio:generated
      
      - name: Check for changes
        run: git diff --exit-code generated/ || echo "Stubs out of date"
      
      - name: Run linters on generated code
        run: |
          pip install pylint
          pylint generated/**/*.py --disable=all --enable=E,F

This ensures stubs are always up-to-date and prevents accidental commits of stale generated code.

Debugging Common Generation Issues​

Import Error: No module named 'orders_pb2'

Verify the generated file exists and is in sys.path:

import sys
print(sys.path)
# Add generated directory if missing
sys.path.insert(0, "generated")

Error: Unknown import symbol: common.types_pb2

Ensure the proto import path and include flag match. If common/types.proto is in protos/common/, use -I protos and compile all files together.

Regenerated code differs from before (false-positive diffs)

protoc output may vary with compiler version. Always use the same grpc-tools version across your team. Lock the version in requirements.txt:

grpcio==1.62.0
grpcio-tools==1.62.0

Key Takeaways​

  • Compile protos with python -m grpc_tools.protoc using -I for imports and --python_out / --grpc_python_out for output directories.
  • Generated files include *_pb2.py (messages) and *_pb2_grpc.py (stubs). Never edit these files; regenerate them when protos change.
  • Use --grpc_python_out=grpc_asyncio: for async stubs suitable for high-concurrency services (recommended in 2026).
  • Organize output to match proto directory structure; add generated/__init__.py to make it a Python package.
  • Automate compilation in Makefiles or CI/CD; never commit stale generated code.

Frequently Asked Questions​

Should I commit generated code to version control?​

No. Generated code is deterministic; commit only .proto files and your Makefile/build script. In CI, regenerate stubs on every build. This prevents merge conflicts and ensures stubs are always in sync with proto definitions.

Can I customize generated code (add methods, change behavior)?​

No. Never edit *_pb2.py or *_pb2_grpc.py. Instead, subclass the servicer or wrap the stub:

# Extend servicer with custom logic
class OrderServiceImpl(OrderServiceServicer):
    def CreateOrder(self, request, context):
        # Custom validation or logging
        return super().CreateOrder(request, context)

What's the difference between sync and async stubs?​

Sync stubs block the calling thread on each RPC; async stubs use async/await and are suitable for high-concurrency servers (100+ concurrent requests). For new code, use --grpc_python_out=grpc_asyncio:generated.

How do I handle backward compatibility when regenerating?​

Protobuf guarantees backward compatibility: old clients work with new servers and vice versa. Regenerating stubs doesn't break compatibility as long as you follow proto versioning rules (never reuse field numbers, never change field types).

Can I use protoc-gen-validate or other plugins?​

Yes, gRPC supports third-party plugins. For validation, use --validate_out= with protoc-gen-validate. For API documentation, use --doc_out=. These run alongside the default generators.

Further Reading​