CLI option `--name` can be used to define the name to identify a
command. If not passed, then the command itself is used. Commands
and names are paired in the same order: The first command executed gets
the first name passed as option.
Close#326
This patch permits the `-L`/`--parameter-list` argument to appear
multiple times. Each additional parameter increases the dimensionality
of the parameter matrix. One benchmark will be run for each combination
of parameters (i.e., the benchmark space is the Cartesian product of the
parameter spaces).
For now, `--parameter-list` and `--parameter-scan` are still mutually
exclusive. If desired, a follow-up change could similarly permit
multiple occurrences of `--parameter-scan` and also permit use of both
listed and scanned parameters together. (After all, `--parameter-scan`
can be thought of as syntactic sugar for a parameter list.)
This implementation is a little profligate with memory usage for
`BenchmarkResult`s: each result contains a separate copy of all
parameter names (as `String`s). This could be done away with by
threading lifetimes through the `BenchmarkResult`s, or by ref-counting
the names.
Because `BenchmarkResult`s now have maps with arbitrary key sets, we can
no longer use `serde` to serialize them to CSV. But a CSV writer is easy
to write by hand, so we just do that.
Fixes#253.
Test Plan:
Some unit tests included. As an integration test, try:
```
cargo run --release -- --runs 10 \
-L foo foo1,foo2 -L bar bar9,bar8 \
'echo {foo} {bar}' \
'printf "%s\n" {foo} {bar}' \
--export-csv /tmp/out
```
with all the export formats.
wchargin-branch: param-list-matrix
wchargin-source: 11051ab222c9de0fbc6ac6a080fbca48e05996c2
This patch adds any parameter used to data output files (currently JSON
and CSV files). With that users have direct access to both input values
(i.e., benchmark parameters) and the resulting timings which should
simplify follow-up analyses.
Allow the user to set the shell used for executing benchmark commands.
The option (-S/--shell) can be overridden, when there are duplicate
options the last option's value will be used.
Strip the empty lines within the results for each benchmark, but keep
the empty line between each benchmark, and before the summary.
Add `color` style option to show color but not be interactive (no
progress bar).
This change makes it possible to forward the stdout of the command the
user tries to benchmark to hyperfine's stdout using the `--print-stdout`
flag.
This should be irrelevant for most use-cases, however it makes it
possible to get some crude benchmarks of terminal emulators similar to
what's currently often done with `time cat file`.
Jordi Chauzi