Build system

Build system#

This page illustrates the main infrastructure behind Espresso - the Espresso Machine.

Build system#

We use scikit-build as Espresso’s packaging tool. This choice is due to the necessity of including compiled code in Espresso.

Like other modern Python packages, we use a pyproject.toml file to specify package metadata.

Build script#

A built Espresso package is a combination of four components:

  • The Espresso core (src/espresso/)

  • All / selected contributed problems (contrib/)

  • The Espresso Machine itself (espresso_machine/)

  • Package metadata files (pyproject.toml and CMakeLists.txt)

Build and validation steps are listed in the Appendix section of the contributor guide. The build script basically copies the files from the above 4 sources into _esp_build, and runs pip install . from _esp_build/ folder. It does perform slight changes to the files:

  1. Versioning template configured in pyproject.toml is modified. Look at the Dynamic versioning section below for details.

  2. A prefix of _ is added to all contribution folder names and Python file names to ensure cleaner tab completion results (e.g. from a iPython console)

  3. All example classes and names are collected into src/espresso/capabilities.py

  4. All examples are run by the build script with a timeout setting of 1 second per method, to quickly build a capability matrix into src/espresso/capabilities.py

  5. All examples with CMakeLists.txt files are recognized as having something to build so the corresponding folders are added to the top-level CMakeLists.txt file

Implementation wise,

  • Most steps of the build script use Python’s inbuilt subprocess, os, sys and shutil libraries

  • The capability matrix is built with the json library

  • versioningit is used to generate version dynamically from git tags

  • All of the command line arguments are parsed by Python’s inbuilt argparse, and the parser of all build-related scripts are defined only once in the file espresso_machine/build_package/_utils.py

The GitHub branch esp_build is reserved for holding the latest package source. Check out the CI / CD page for more details.

Validation script#

Again, we refer readers to the Appendix section of the contributor guide for a list of steps in the validation script.

When you run

$ python espresso_machine/build_package/validate.py --pre

It’s equivalent to

$ python espresso_machine/build_package/test_examples.py --pre

And when you run

$ python espresso_machine/build_package/validate.py --post

It’s equivalent to

$ python espresso_machine/build_package/test_examples.py --post
$ python espresso_machine/build_package/check_requires.py

Implementation wise,

  • pytest is used to invoke and report all the validation job

  • The pytest fixture feature is extensively used to parameterise the validation of different problem examples

  • Top level command lines are still parsed by the parser defined in _utils.py, but pytest specific files (test_examples.py and check_requires.py) extracts command line information from the parser defined in conftest.py instead

Dynamic versioning#

We use versioningit to generate dynamic version from the output of git describe, and write the dynamic version into file src/espresso/_version.py (which is hidden from version control).

When we run pip install . from the root level of this project, the generated version always ends with .core. This behaviour is configured by the versioningit_config variable in espresso_machine/versioning/__init__.py.

When we run pip install . from _esp_build/ folder, the generated version won’t end with .core because we use the configurations written in espresso_machine/versioning/versioning_for_full_package.py instead.