Python

Python#

This Chapter has some advanced concepts when working with Python.

Code Wrapping#

Python is able to run code written in many other languages, provided they provide a suitable interface or ‘wrapper’. In particular it is common for C or C++ code to be wrapped with Python.

There are several approaches that can be taken with wrapping, the most basic is to use the built-in ctypes Foreign Function library. This is useful if your library has a basic C interface and you are comfortable with low-level APIs.

The cffi library also targets C APIs and provides a higher level interface, although it requires an extra third-party module dependency.

Cython is a Python language superset with extra functionality for C/C++ interactions. The interaction code is similar to Python.

SWIG is a general wrapping interface for C and C++, supporting many target languages at once.

nanobind is a wrapping library focused on Python and C++, it allows wrapping of advanced interfaces and data types with high performance. It is a successor of pybind11, which itself built on many of the ideas of Boost Python.

The ichello ICHEC project is an end-to-end example of Python wrapping with nanobind.

Packaging#

Packaging wrapped code is more involved than native or vanilla Python, since compilation of the C/C++ code needs to be managed and binary redistributable packages need to be created for all supported systems, rather than a single source redistributable.

The CMake build system is often used to manage building the C and C++ code. This can be driven via a basic setup.py script pointed to by the pyproject.toml Python project description, however it may be easier to use a dedicated tool for wrapped Python projects. scikit-build-core, a recent successor to Scikit Build is becoming popular for this purpose. It can be used as a build-system in the pyproject.toml and will automatically handle CMake integration and binary generation.

In addition to the ichello ICHEC project the Nanobind Official Scikit Build example may be a useful reference for getting started.

Development#

Combined development of the Python and C/C++ code can be awkward. If a project is primarily focused on the C/C++ element it may be worth pulling it in from a separate repo - for example as a CMake External Project and keeping only interfacing code in the Python repo.

If you are working on or troubleshooting a combined Python and C++ project you will likely want to use an editrable install without build isolation - meaning you will need to install dependency packages explicitly as follows:

pip install -e . --no-build-isolation

Debugging and Profiling#

Debugging wrapped code can be tricky. If using the lldb debugger on Mac and a Python virtual environment you will need to set settings set target.process.stop-on-exec false to avoid it exiting early if Python interpreters are called in a chain.

Futher Reading#

Building an Interpreter#

Some scenarios where you might end up building a Python interpreter include:

  • Building Python bindings on a system without root and missing Python headers

  • Distributing an application with a bundled interpreter

  • Advanced performance analysis or debugging

CPython is the most typical interpreter to build. To do so you can do:

git clone https://github.com/python/cpython.git

You probably want to check out a branch corresponding to a release version:

cd cpython
git checkout 3.9

Then you can do:

./configure --prefix $INSTALL_DIR --enable-shared
make -j 4 
make install

installing in $INSTALL_DIR otherwise you will break your system. You can look at configure --help for other options that may be of interest - such as enabling profiling.

Contents