Add a --setup option for the use-case of wanting to do one-off setup
before a set of benchmarks, not the once-per-test setup that --prepare
does. This is useful for the cases noted in the updated documentation.
Per the feedback on https://github.com/sharkdp/hyperfine/pull/448 this
new "--setup" option will steal the "-s" short option from
"--style" (initially this was called "--build" and used the
non-conflicting "-b").
Potential future work:
I'd prefer if this and --cleanup took N number of commands, so you
could do e.g.:
hyperfine -L rev master,next \
-b 'git worktree add /tmp/test-{r}' \
-b 'make -C /tmp/test-{r} all' \
'make -C /tmp/test-{r} test' \
-c 'git worktree remove /tmp/test-{r}'
I.e. a shortcut around not providing these with &&, which makes things
a bit more readable.
But the --cleanup option doesn't do that, so let's just go with what
it's doing for consistency, so for this you'll now need to do:
hyperfine -L rev master,next \
-b 'git worktree add /tmp/test-{r} &&
make -C /tmp/test-{r} all' \
'make -C /tmp/test-{r} test' \
-c 'git worktree remove /tmp/test-{r}'
Extend the --runs=N option added in d78c33b (Added options to specify
the max/exact numbers of runs., 2018-09-09) to support --runs=1
instead of dying with a usage error.
This is useful to do ad-hoc testing of your commands while they might
still have syntax errors, or just for doing one run where you don't
care about the stddev.
Before:
$ /usr/bin/hyperfine -s basic -r 1 -L n 5,10 'sleep 0.{n}'
Error: Number of runs below two
After (-s basic) also works:
$ hyperfine -r 1 -L n 5,10 'sleep 0.{n}'
Benchmark 1: sleep 0.5
Time (abs ≡): 500.6 ms [User: 0.6 ms, System: 0.0 ms]
Benchmark 2: sleep 0.10
Time (abs ≡): 100.8 ms [User: 0.7 ms, System: 0.0 ms]
Summary
'sleep 0.10' ran
4.97 ± 0.00 times faster than 'sleep 0.5'
This likewise combines correctly with -m and -M, probably not very
useful, but if you're tweaking an existing command-line:
$ hyperfine -m 1 -M 1 -L n 5,10 'sleep 0.{n}'
Benchmark 1: sleep 0.5
Time (abs ≡): 500.6 ms [User: 0.5 ms, System: 0.0 ms]
Benchmark 2: sleep 0.10
Time (abs ≡): 100.6 ms [User: 0.6 ms, System: 0.0 ms]
Summary
'sleep 0.10' ran
4.98 ± 0.00 times faster than 'sleep 0.5'
The "± 0.00" output in "faster than" should probably be adjusted too,
or we could keep it for consistency. I didn't implement that because
this is the first time I do anything in Rust, and I ran out of
template to copy when wanting to quickly implement this in
write_benchmark_comparison() in main.rs.
Shows an error message if the argument to options like '--warmup <N>'
can not be parsed:
> hyperfine --warmup xxx "sleep 1"
Error: Could not read numeric argument to '--warmup': invalid digit found in string
closes#337
CLI option `--name` can be used to define the name to identify a
command. If not passed, then the command itself is used. Commands
and names are paired in the same order: The first command executed gets
the first name passed as option.
Close#326
This patch permits the `-L`/`--parameter-list` argument to appear
multiple times. Each additional parameter increases the dimensionality
of the parameter matrix. One benchmark will be run for each combination
of parameters (i.e., the benchmark space is the Cartesian product of the
parameter spaces).
For now, `--parameter-list` and `--parameter-scan` are still mutually
exclusive. If desired, a follow-up change could similarly permit
multiple occurrences of `--parameter-scan` and also permit use of both
listed and scanned parameters together. (After all, `--parameter-scan`
can be thought of as syntactic sugar for a parameter list.)
This implementation is a little profligate with memory usage for
`BenchmarkResult`s: each result contains a separate copy of all
parameter names (as `String`s). This could be done away with by
threading lifetimes through the `BenchmarkResult`s, or by ref-counting
the names.
Because `BenchmarkResult`s now have maps with arbitrary key sets, we can
no longer use `serde` to serialize them to CSV. But a CSV writer is easy
to write by hand, so we just do that.
Fixes#253.
Test Plan:
Some unit tests included. As an integration test, try:
```
cargo run --release -- --runs 10 \
-L foo foo1,foo2 -L bar bar9,bar8 \
'echo {foo} {bar}' \
'printf "%s\n" {foo} {bar}' \
--export-csv /tmp/out
```
with all the export formats.
wchargin-branch: param-list-matrix
wchargin-source: 11051ab222c9de0fbc6ac6a080fbca48e05996c2
Clemens Wasser