This section explains the development, integration and distritbuion process of the Benchmarking Suite. Intended readers are developers.

Continuous Integration


Release Steps

Checklist to release the Benchmarking Suite

  1. Commit any not-committed file on the workspace
  2. Identify whcih modules needs to be released (see commits since the latest release, see the changelog)
  3. Update the changelog file if not done (use messages in the commits as reference)

Modules Release

For each module to release:

  1. increase the version number in the __init__py file

  2. create the source distribution package and upload on PYPI Testing (remove the -r pypitest to upload on the official PYPI)

    python sdist upload -r pypitest
  3. to test the release from PYPI test:

    # create a new virtual env
    virtualenv -p /usr/bin/python3.5 venvXX
    # activate the virtualenv
    source venvXX/bin/activate
    # install the modules to test
    pip install -v -i --extra-index-url -U benchsuite.core
  4. upload the distribution packages on PYPI

    python sdist upload
  1. commit and push everything on GitHub
  2. create a release on GitHub (this will also create a tag)

Milestone Release

  1. Check all the modules and the versions that will be included. Release modules if necessary
  1. In benchsuite-docs, update the version in
  2. Update the changelog.rst with the changelog for this milestone
  1. Commit the documentation on GitHub and create a tag. This will also create a new tag in readthedocs
  2. Update the “.release” Dockerfiles in benchsuite-docker project
  3. Commit benchsuite-docker and create a new tag. This will trigger the creation of a new tag for docker images


Documentation is automatically built on ReadTheDocs at every commit


Docker containers are built automatically from Dockerfiles located in the benchsuite-docker repository.

To create a new tag of Docker images, create a tag in the Git repository that starts with “v” (e.g. “v2.0”, “v1.2.3”, “v1.2.3-beta1”, …)