Tailleur

A generic benchmark framework and runner.

This project aims to provide tools for blackbox benchmarking, with options to drop into other tools for microbenchmarking.

Running

From source:

git clone https://github.com/efficios/tailleur.git && cd tailleur
poetry install
poetry run tailleur

Configuration

Defaults

A JSON or YAML file

---
# config.yaml
config:
  runs: 10   # The number of runs for each benchmark
search_paths:
  - /path/to/x
  - relative/path/to/x

Set the defaults via command-line:

poetry run tailleur --config /path/to/config.yaml

Configuration keys may also be overridden on the command-line as follows:

# Set runs to '1'
poetry run tailleur --config-key runs=1

# Set a nested value
poetry run tailleur --config-key my.nested.path=value

Benchmark selection

via command line

The -f, --filter argument may be used to filter any selected or discovered benchmarks. By default, the argument is interpreted as a benchmark name, but the type of filter module maybe specified explicitly.

The following modules are available:

• name (default)
• re
• tag

Matches may be inverted by prefixing the filter argument with !. Take care to single-quote the argument in shells where ! is a control character.

Example 1: Filter for benchmarks whose name is Throughput

# Matches module_x.Throughput and module_y.Throughput
poetry run tailleur -f Throughput
# or,
poetry run tailleur -f name:Throughput

Example 2: Filter for benchmarks whose module and name match explicitly:

poetry run tailleur -f module_x.Throughput

Example 2: Filter benchmarks whose name matches a regex:

# Filter for all benchmarks in module_y.
poetry run tailleur -f 're:module_y\..*'

Example 3: Filter benchmarks with a given tag:

# Filter for all benchmarks with the tag `slow`
poetry run tailleur -f tag:slow

Example 4: Filter for benchmarks whose name is not Throughput:

poetry run tailleur -f '!name:Throughput'

via configuration file

A JSON or YAML file

---
# suite.yaml
search_paths:
  - /path/to/x
  - relative_to_cwd/x

benchmarks:
  - name: [module.]ClassName
    # Configuration is merged with the defaults
    config:
      runs: 2
    params:
      - # set1
        param_X: vX
        param_Y: vY
      - # set2
        param_X: vX_2
        param_Y: vY_2

Specify it as follows:

poetry run tailleur --benchmarks suite.yaml

Why

1. Existing benchmarking tools in Python such as ASV and pytest-benchmark are meant to benchmark Python projects.
2. Benchmarks should be able to return more metrics than just execution time.
3. Benchmark results should include more metadata on the running environment.

References and more

• ASV
• pytest-benchmark
• google/becnhmark
• hyperfine