Build system

This page illustrates the main infrastructure behind Espresso - the Espresso Machine.

Build system

We use scikit-build as Espresso’s packaging tool. This choice is due to the necessity of including compiled code in Espresso.

Like other modern Python packages, we use a pyproject.toml file to specify some package metadata. Additionally, we keep the traditional setup.py file for most of other meta information. It would be cleaner to have only one pyproject.toml file to host everything, but scikit-build doesn’t support this approach yet.

Build script

A built Espresso package is a combination of four components:

• The Espresso core (src/espresso/)

• All / selected contributed problems (contrib/)

• The Espresso Machine itself (espresso_machine/)

• Package metadata files (setup.py, CMakeLists.txt and pyproject.toml)

Build and validation steps are listed in the Appendix section of the contributor guide. The build script basically copies the files from the above 4 sources into _esp_build, and runs pip install . from _esp_build/ folder. It does perform slight changes to the files:

1. Versioning template configured in setup.py is modified. Look at the Dynamic versioning section below for details.

2. A prefix of _ is added to all contribution folder names and Python file names to ensure cleaner tab completion results (e.g. from a iPython console)

3. All example classes and names are collected into src/espresso/capabilities.py

4. All examples are run by the build script with a timeout setting of 1 second per method, to quickly build a capability matrix into src/espresso/capabilities.py

5. All examples with CMakeLists.txt files are recognized as having something to build so the corresponding folders are added to the top-level CMakeLists.txt file

Implementation wise,

• Most steps of the build script use Python’s inbuilt subprocess, os, sys and shutil libraries

• The capability matrix is built with the json library

• versioningit is used to generate version dynamically from git tags

• All of the command line arguments are parsed by Python’s inbuilt argparse, and the parser of all build-related scripts are defined only once in the file espresso_machine/build_package/_utils.py

The GitHub branch esp_build is reserved for holding the latest package source. Check out the CI / CD page for more details.

Validation script

Again, we refer readers to the Appendix section of the contributor guide for a list of steps in the validation script.

When you run

$ python espresso_machine/build_package/validate.py --pre

It’s equivalent to

$ python espresso_machine/build_package/test_examples.py --pre

And when you run

$ python espresso_machine/build_package/validate.py --post

It’s equivalent to

$ python espresso_machine/build_package/test_examples.py --post
$ python espresso_machine/build_package/check_requires.py

Implementation wise,

• pytest is used to invoke and report all the validation job

• The pytest fixture feature is extensively used to parameterise the validation of different problem examples

• Top level command lines are still parsed by the parser defined in _utils.py, but pytest specific files (test_examples.py and check_requires.py) extracts command line information from the parser defined in conftest.py instead

Dynamic versioning

We use versioningit to generate dynamic version from the output of git describe, and write the dynamic version into file src/espresso/_version.py (which is hidden from version control).

When we run pip install . from the root level of this project, the generated version always ends with .core. This behaviour is configured by the versioningit_config variable in setup.py.

When we run pip install . from _esp_build/ folder, the generated version won’t end with .core because the build script removes .core from that configuration.