Skip to main content

Building Cython Extensions: Compilation and Distribution

Writing a fast Cython function is the easy part; shipping it to users is the challenge. You must compile for multiple platforms (Linux, macOS, Windows), Python versions, and architectures. setuptools and wheel handle this complexity. This article teaches the build system that powers SciPy, scikit-learn, and pandas.

Project Structure for Cython​

A typical Cython project layout:

myproject/
├── setup.py
├── setup.cfg
├── pyproject.toml
├── MANIFEST.in
├── src/
│   └── myproject/
│       ├── __init__.py
│       └── fast_module.pyx
└── tests/
    └── test_fast_module.py

Minimal setup.py for Cython​

Create a setup.py that compiles your .pyx files:

from setuptools import setup, Extension
from Cython.Build import cythonize

extensions = [
    Extension(
        "myproject.fast_module",
        ["src/myproject/fast_module.pyx"],
    )
]

setup(
    name="myproject",
    version="0.1.0",
    author="Your Name",
    description="Fast matrix operations with Cython",
    ext_modules=cythonize(extensions),
    package_dir={"": "src"},
    packages=["myproject"],
)

Build in-place:

pip install -e .

This installs the package in development mode, compiling Cython files.

Advanced setup.py with Compiler Flags​

Production builds need optimization flags. Add extra_compile_args:

from setuptools import setup, Extension
from Cython.Build import cythonize

extensions = [
    Extension(
        "myproject.fast_matrix",
        ["src/myproject/fast_matrix.pyx"],
        extra_compile_args=['-O3', '-march=native'],  # High optimization
        extra_link_args=[],
    )
]

setup(
    name="myproject",
    version="0.1.0",
    ext_modules=cythonize(extensions, compiler_directives={
        'language_level': '3',
        'boundscheck': False,
        'wraparound': False,
    }),
)

Compiler directives (boundscheck, wraparound) disable safety checks for speed (use only after testing).

Using pyproject.toml for Modern Python Packaging​

pyproject.toml is the modern way to define build requirements. Create one:

[build-system]
requires = ["setuptools", "cython>=0.29.36"]
build-backend = "setuptools.build_meta"

[project]
name = "myproject"
version = "0.1.0"
description = "Fast Cython extensions for Python"
authors = [
    {name = "Your Name", email = "[email protected]"}
]
requires-python = ">=3.8"

[tool.cython]
language_level = "3"

[tool.cython.compilation]
compiler_directives = {boundscheck = false, wraparound = false}

Building Wheels for Distribution​

Wheels are pre-compiled Python packages. Users install with pip without building:

pip install build
python -m build --wheel

This creates dist/myproject-0.1.0-cp310-cp310-linux_x86_64.whl (a pre-compiled binary for Python 3.10 on Linux 64-bit).

Share wheels on PyPI:

pip install twine
twine upload dist/myproject-*.whl

Users then install via:

pip install myproject

Building for Multiple Python Versions and Platforms​

To support Python 3.8–3.12 on Linux, macOS, and Windows, use cibuildwheel:

pip install cibuildwheel

Create .github/workflows/build.yml:

name: Build wheels

on: [push, pull_request]

jobs:
  build:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
    steps:
      - uses: actions/checkout@v3
      - uses: pypa/[email protected]
        env:
          CIBUILDWHEEL_SKIP: "pp*"  # Skip PyPy
      - uses: actions/upload-artifact@v3
        with:
          path: ./wheelhouse/*.whl

Push to GitHub; cibuildwheel automatically builds wheels on 12+ platforms.

Testing Compiled Extensions​

Test your Cython code before distribution:

# tests/test_fast_module.py
import pytest
from myproject.fast_module import matrix_multiply

def test_matrix_multiply():
    import numpy as np
    A = np.array([[1.0, 2.0], [3.0, 4.0]])
    B = np.array([[5.0, 6.0], [7.0, 8.0]])
    result = matrix_multiply(A, B)
    expected = np.array([[19.0, 22.0], [43.0, 50.0]])
    np.testing.assert_array_almost_equal(result, expected)

Run tests:

pytest tests/

Troubleshooting Common Build Issues​

Issue: error: Microsoft Visual C++ 14.0 is required (Windows)

Solution: Install Microsoft C++ Build Tools or Visual Studio Community.

Issue: ImportError: No module named 'fast_module'

Solution: You compiled out-of-place; reinstall in development mode:

pip install -e .

Issue: undefined reference to '__gfortran_*' (BLAS/LAPACK)

Solution: Link BLAS/LAPACK libraries in setup.py:

Extension(
    "myproject.linalg",
    ["src/myproject/linalg.pyx"],
    libraries=["blas", "lapack"],
)

Distributing on PyPI​

  1. Create a PyPI account at https://pypi.org.
  2. Create a .pypirc file in your home directory:
[distutils]
index-servers =
    pypi

[pypi]
repository = https://upload.pypi.org/legacy/
username = __token__
password = pypi-AgEIcHlwaS5vcmc...
  1. Build and upload:
python -m build
twine upload dist/*

Users can now install:

pip install myproject

Performance Tips for Distribution​

  • Use wheels, not source distributions: Wheels are pre-compiled; users don't pay compilation overhead.
  • Test on multiple Python versions: Cython code compiled for Python 3.8 doesn't work on 3.12; rebuild for each version.
  • Specify compiler optimization flags: Use -O3 and -march=native for production; document this in README.
  • Include .pyx source in wheels: Optional but good for debugging; use MANIFEST.in.

Key Takeaways​

  • Use setuptools and cythonize() to define Cython extensions in setup.py.
  • Compile with pip install -e . for development.
  • Build wheels with python -m build --wheel for distribution.
  • Use cibuildwheel and GitHub Actions to automate multi-platform builds.
  • Test thoroughly before uploading to PyPI.

Frequently Asked Questions​

Why do I need setuptools if Cython has its own compiler?​

Cython compiles .pyx to .c, but you still need a C compiler and linker. setuptools handles that integration, linking dependencies, and platform-specific flags.

Can I distribute Cython source code (.pyx) instead of wheels?​

Yes, but it's slower. Users must compile on install (requires a C compiler). For mass distribution, wheels are better.

What's the difference between a wheel and an sdist (source distribution)?​

  • Wheel (.whl): Pre-compiled binary; instant install.
  • Sdist (.tar.gz): Source code; compiled on user's machine (slow, requires C compiler).

PyPI stores both; pip prefers wheels.

How do I handle platform-specific code in Cython?​

Use sys.platform checks in Python code, or preprocessor directives in .pyx:

# platform-specific import
IF UNAME_SYSNAME == "Windows":
    from libc.stdint cdef extern from "windows.h":
        pass
ELSE:
    from libc.unistd cdef extern from "unistd.h":
        pass

Does wheel size matter for distribution?​

Smaller is better; wheels travel over the network and consume user disk space. Use strip to remove debug symbols; on Linux:

strip dist/*.so

Further Reading​