Command line tool¶

Install¶

The Benchmarking Suite Command line tool can be installed with:

pip install benchsuite.cli

If the installation was successful, the benchsuite command should be in your path.

Usage and Examples¶

Create a new session¶

To create a new session, the Benchmarking Suite needs two information: the provider configuration and the service type. The command line tool offers multiple options to specify these parameters.

Provider Configuration

There are different alternatives:

specify a name with the --provider option (e.g. --provider myamazon). In this case, a provider configuration file named <name>.conf will be searched in the configuration path;
specify a filename in the --provider option (e.g. --provider /path/myamazon.conf). The configuration file specified will be used;
store the provider configuration in the BENCHSUITE_PROVIDER environment variable.

As example, the following invocations load the same provider configuration:

$ benchsuite new-session --provider my-amazon.conf ...

$ benchsuite new-session --provider $BENCHSUITE_CONFIG_FOLDER/providers/my-amazon.conf ...

$ export BENCHSUITE_PROVIDER=```cat $BENCHSUITE_CONFIG_FOLDER/providers/my-amazon.conf```
$ benchsuite new-session ...

Service Type

The service type can be specified using the --service-type option (e.g. --service-type ubuntu_micro). The value of the service type must be one of the ones defined in the provider configuration. Alternatively the ``BENCHSUITE_SERVICE_TYPE` enviornment variable can be used.

If neither the --service-type option nor the ``BENCHSUITE_SERVICE_TYPE` enviornment variable are specified and the provider configuration defines only ONE service type, that one will be used, otherwise the invocation will fail.

Documentation¶

This is an autogenerated documentation from the Python argparse options.

usage: benchsuite [-h] [--verbose] [--quiet] [--config CONFIG]
                  {shell,new-session,new-exec,prepare-exec,run-exec,list-sessions,list-providers,list-benchmarks,destroy-session,list-execs,collect-exec,multiexec}
                  ...

Named Arguments¶

–verbose, -v

print more information (3 levels)

–quiet, -q

suppress normal output

Default: False

–config, -c

foo help

Sub-commands:¶

shell¶

Starts an interactive shell

benchsuite shell [-h]

new-session¶

Creates a new benchmarking session

benchsuite new-session [-h] [--provider PROVIDER]
                       [--service-type SERVICE_TYPE] [--property PROPERTY]
                       [--user USER] [--tag TAG]

Named Arguments¶

`–provider, -p`	The name for the service provider configuration or the filepath of the provider configuration file. Alternatively, the provider configuration can be specified in the environment variable BENCHSUITE_PROVIDER (the content of the variable must be the actual configuration not the filepath)
`–service-type, -s`
	The name of one of the service types defined in the provider configuration. Alternatively, it can be specified in the BENCHSUITE_SERVICE_TYPE environment varaible
`–property, -P`	Add a user defined property to the session. The property must be expressed in the format <name>=<value>
`–user, -u`	sets the “user” property. It is a shortcut for “–property user=<name>
`–tag, -t`	sets one or more session tags. Internally, tags are stored as properties

new-exec¶

Creates a new execution

benchsuite new-exec [-h] session tool workload

Positional Arguments¶

`session`	a valid session id
`tool`	a valid benchmarking tool
`workload`	a valid benchmarking tool workload

prepare-exec¶

Executes the install scripts for an execution

benchsuite prepare-exec [-h] id

Positional Arguments¶

`id`	a valid id of the execution

run-exec¶

Executes the execute scripts for an execution

benchsuite run-exec [-h] [--storage-config STORAGE_CONFIG] [--async] id

Positional Arguments¶

`id`	a valid id of the execution

Named Arguments¶

–storage-config, -r

Specify a custom location for the storage configuration file

–async

start the execution of the scripts and return (do not wait for the execution to finish)

Default: False

list-sessions¶

a help

benchsuite list-sessions [-h]

list-providers¶

a help

benchsuite list-providers [-h]

list-benchmarks¶

a help

benchsuite list-benchmarks [-h]

destroy-session¶

a help

benchsuite destroy-session [-h] id

Positional Arguments¶

id

bar help

list-execs¶

lists the executions

benchsuite list-execs [-h]

collect-exec¶

collects the outputs of an execution

benchsuite collect-exec [-h] id

Positional Arguments¶

`id`	the execution id

multiexec¶

Execute multiple tests in a single benchmarking session

benchsuite multiexec [-h] [--provider PROVIDER] [--service-type SERVICE_TYPE]
                     [--storage-config STORAGE_CONFIG] [--property PROPERTY]
                     [--user USER] [--tag TAG]
                     tests [tests ...]

Positional Arguments¶

tests

one or more tests in the format <tool>[:<workload>]. If workload is omitted, all workloads defined for that tool will be executed

Named Arguments¶

`–provider, -p`	The name for the service provider configuration or the filepath of the provider configuration file
`–service-type, -s`
	The name of one of the service types defined in the provider configuration. If not specified, all service types will be used
`–storage-config, -r`
	Specify a custom location for the storage configuration file
`–property, -P`	Add a user defined property to the session. The property must be expressed in the format <name>=<value>
`–user, -u`	sets the “user” property. It is a shortcut for “–property user=<name>
`–tag, -t`	sets one or more session tags. Internally, tags are stored as properties