Create new example

👋 Everyone is welcomed to contribute with their own forward code. We aim to reduce the barrier of contributing so don’t worry if you are not familiar with some technical stuff - we are here to help.

Add your own Espresso problem

1. A new contribution to Espresso has to conform to a consistent file structure. The simplest way to ensure that a new contribution includes all files is to start the creation of a new problem from the template. Run the following command from the the path to Espresso:

   python espresso_machine/new_contribution/create_new_contrib.py <problem-name>
   

   Replacing problem-name with your Espresso problem name in snake case (e.g. gravity_density, polynomial_regression). Remember to choose a sensible, unique name for your contribution that makes it easy to understand what the example is about.

2. Navigate to folder <path-to-espresso>/contrib/<problem-name>, and you’ll see template files needed for a new contribution. Each Espresso example is organised around a central class object that contains, at minimum, a set of functions with names that are shared by all examples.

   

3. Now the folder is all yours, get started with the following checklist.

   Checklist and tips

   • contrib/<problem-name>/README.md

     • Document anything you’d like to add for this problem, such as the what the problem is about and some brief intro of the theory behind.

     • This will be automatically rendered into Test Problems Reference.

   • contrib/<problem-name>/LICENCE

     • We use a 2-clause BSD licence as the default one. Feel free to replace it with a licence that suits you best.

   • contrib/<problem-name>/<problem-name>.py

     • The development will be centered around the autogenerated class ProblemName. It is a subclass of EspressoProblem so you might find the API reference helpful.

     • All standard attributes have been declared in the template but are left for you to implement. You’ll see that some are required (with TODO) and others are optional.

     • If you would like to load data from files, please use our utility functions to get the absolute path before calling your loading function.

     • Apart from the standard attributes, there are many more functions and values that a new contribution can contain, for examples:

       • inversion_suggestion: a string containing inversion suggestions

       • reg_param_suggestion: a sensible value for regularization parameter

       • dx: spatial resolution in x-direction

       • dt: temporal resolution

       • nt: number of time steps

       • The possibilities are endless! Whatever information you find helpful is probably also helpful for the users. Simply attach them to the problem class as additional attributes.

     • We aim to follow Python PEP8 style conventions to make source code readable. Though not strictly enforced, we recommend Black formatter and PyLint to maintain a good coding style.

   • setup.py, in particular the INSTALL_REQUIRES variable, if needed.

4. Test running your code.

   Troubleshooting: issue with relative import

   Note that you might see an error if you have any relative import in the main Python file:

   # file: <problem-name>.py
   
   from .lib import *
   

   In this case, use contrib as your working directory and import your contribution in the following example way:

   $ pwd                                        # check you are in the right folder
   <path-to-espresso>/contrib
   $ python
   >>> from example_name import ExampleName     # ...and import this way
   

   Or the following example if you are running a file:

   # file: contrib/tmp.py                       # create tmp file in the right folder
   from example_name import ExampleName         # ...and import this way
   

5. Validate your code with Espresso by running:

   $ python espresso_machine/build_package/build.py --pre --post -c <example_name>
   

   which includes the following steps:

   • Validate all required and standard attributes that you’ve implemented

   • Build your contribution into a temporary source code folder _esp_build

   • Install your local development Espresso version

   • Validate all required and standard attributes from the installed package

   • Check that packages needed to run your code are listed by Espresso

   Read on the appendix sections in this page for how an Espresso example is validated and how Espresso is built, and continue with the Developer Guide further details the whole infrastructure (i.e. the espresso_machine).

Jupyter Notebook

Additionally, we encourage you to add a Jupyter Notebook with an identical name into the folder Jupyter Notebooks that contains the following:

1. An extensive description of the new Espresso Problem, containing information about (but not limited to):

   • the forward calculation (ie. the underlying physics) and how it was implemented.

   • which inversion method is used (and regularisation) and how it was implemented.

   • the physical unit of relevant variables, but at least of model and data.

   • all changeable parameters, possibly in a list.

2. An example of the new problem being used, with a reasonable output.

Appendix I: build steps

Usage:

$ python build.py [--pre] [--post] [--no-install] [-c <example_name>] [--file <file_name>]

For instance, to install your development version locally, run the following in your terminal:

$ python espresso_machine/build_package/build.py

Run the following for detailed usage information:

$ python espresso_machine/build_package.build.py --help

The following table describes what happens when we package Espresso:

How Espresso is packaged

Step	What’s done	How it’s done
1	Clean _esp_build/	shutil.rmtree
2	Move meta data files to _esp_build/	shutil.copy
3	Move src/ content to _esp_build/src/espresso	shutil.copytree
4	Move contrib/ content to _esp_build/src/espresso/ + _esp_build/src/espresso/__init__.py + _esp_build/src/espresso/capabilities.py	shutil.copytree, a series of file opening and string manipulation
5	Write dynamic version and extra versioningit configs into _esp_build/pyproject.toml	versioningit
6	Install package from _esp_build/	pip install _esp_build

Appendix II: validation steps

Usage:

$ python validate.py [-h] [--all] [--pre] [--post] [--contrib CONTRIBS] [--file <file_name>]

For instance, to test whether your new contribution aligns with the Espresso standard, run the following in your terminal:

$ python espresso_machine/build_package/validate.py -c <contrib-name>

Or the following for a complete check on all examples (including yours and existing ones for regression test), both before and after Espresso installed:

$ python espresso_machine/build_package/validate.py

Anyway, run the following for a detailed usage of this script:

$ python espresso_machine/build_package/validate.py --help

The following table describes what happens when we validate a certain version of Espresso:

How an Espresso contribution is validated

Step	What’s done	How it’s done
1	Check the contribution folder name matches the main Python file name (contrib/<contrib_name>/<contrib_name>.py)	assert f"{contrib_name}.py" in file_names
2	Check README.md, LICENCE and __init__.py exist	assert required_file in file_names
3	Check the class name is listed in __all__ in file __init__.py	assert contrib_name_class in parent_module.__all__
4	Check the contribution provides access to the required metadata	Pull out the metadata field of the contribution class and check those attributes are of correct types
5	Check required methods / properties are implemented and a complete workflow can run for each example number	Run from example_number=1 up until an exception is raised or reached 100. For each example, try to get model_size, data_size, good_model (flat array like, length = model_size), starting_model (flat array like, length = model_size), data (flat array like, length = data_size); Run forward(model) (output to be flat array like, length = data_size). Where “flat array like” is checked via np.ndim(obj) == 1
6	Check optional methods / properties, if implemented, have the correct type signatures	For each example, check that the outputs of forward(model, return_jacobian=True), jacobian(model) (if implemented) have flat array like synthetics and 2D array like jacobian; Check description (if exists) is string; Check covariance_matrix and inverse_covariance_matrix are in shape (data_size, data_size) and one is the inverse of the other (if implemented); Check plot_model and plot_data (if implemented) return an instance of matplotlib.figure.Figure; Check misfit, log_likelihood and log_prior (if implemented) return float
7	Check LICENCE file is not empty	assert os.stat("LICENCE").st_size != 0