diff --git a/CHANGELOG.md b/CHANGELOG.md index a5ce7af..38912cd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ ## Breaking changes +- Replace `--time-unit`/`-u` with `--metrics METRIC[:UNIT],…`. Select multiple metrics for terminal output; the first metric is primary for all non-JSON exports. Units are chosen automatically for terminal and markup output when omitted. Counters support decimal `k`, `M`, and `B` suffixes. +- Remove `--sort`, `--reference`, and `--reference-name`. Commands and exports always remain in input order, and the first expanded benchmark is the reference for every metric. +- Replace time-only terminal output and the final speed ranking with a per-command metric table. Show wall-clock time and peak resident memory by default (wall-clock time only on Windows), with percentage changes from the first command for subsequent benchmarks. +- Export only the primary metric to CSV, Markdown, AsciiDoc, and org-mode. CSV uses an explicitly selected unit or the metric's base unit when omitted. JSON continues to contain all collected metrics in base units and identifies the primary metric. - Benchmarked commands are now always executed directly, without an intermediate shell (`--shell=none`). Use `-S` (an alias for `--shell=default`) to restore the previous behavior (`sh` on Unix, `cmd.exe` on Windows), or select a shell with `--shell `. - The JSON format for `--export-json` has changed (schema version 2). The new format now includes metadata, per-run measurements, and statistical summaries for all measured quantities, see #790 (@sharkdp). Code that reads those exported JSON files must be updated. For example, `results[i].mean` is now `results[i].summary.time_wall_clock.mean`, and `results[i].times` is replaced by `results[i].measurements[j].time_wall_clock.value`. The new structure looks like this: diff --git a/README.md b/README.md index aca3f9b..54422a5 100644 --- a/README.md +++ b/README.md @@ -5,18 +5,13 @@ A command-line benchmarking tool. -**Demo**: Benchmarking [`fd`](https://github.com/sharkdp/fd) and -[`find`](https://www.gnu.org/software/findutils/): - -![hyperfine](https://i.imgur.com/z19OYxE.gif) - ## Features -* Statistical analysis across multiple runs. +* Statistical analysis across multiple runs, for time, peak memory, and hardware counters. * Support for arbitrary shell commands. * Constant feedback about the benchmark progress and current estimates. * Warmup runs can be executed before the actual benchmark. -* Cache-clearing commands can be set up before each timing run. +* Cache-clearing commands can be set up before each benchmark run. * Statistical outlier detection to detect interference from other programs and caching effects. * Export results to various formats: CSV, JSON, Markdown, AsciiDoc. * Parameterized benchmarks (e.g. vary the number of threads). @@ -41,11 +36,65 @@ number of runs, you can use the `-r`/`--runs` option: hyperfine --runs 5 'sleep 0.3' ``` -If you want to compare the runtimes of different programs, you can pass multiple commands: +If you want to compare different programs, you can pass multiple commands: ```sh hyperfine 'hexdump file' 'xxd file' ``` +Commands run and appear in input order. The first command is the reference for all comparisons; +each subsequent result shows its percentage change from that reference. With parameterized +benchmarks, the first command after parameter expansion is the reference. + +### Choosing metrics and units + +By default, hyperfine displays wall-clock time and peak resident memory +(`time_wall_clock,memory_peak_resident`). On Windows, the default is wall-clock time only, +because peak resident memory is not yet supported. Use `--metrics` to select a +comma-separated list of metrics, with an optional unit after each metric: + +```sh +hyperfine \ + --metrics memory_peak_resident:MiB,time_wall_clock:ms,instructions:B \ + './baseline' './candidate' +``` + +Use `--metrics=default` for the platform defaults or `--metrics=all` for all available +metrics in the order listed below, with wall-clock time primary. Both presets choose display +units automatically and must be used alone, without other metrics or unit suffixes. +The `all` preset skips metrics unsupported by the platform or shell, or missing from any run +of a benchmark. If a metric is unavailable for the reference, its change displays as `N/A`. + +Each completed benchmark displays one row per metric, in the selected order, with its mean, +standard deviation, minimum, and maximum. The first benchmark has no change column. +For subsequent commands, a change of `+50.0%` means +the mean is 50% greater than the reference; `-20.0%` means it is 20% smaller. These percentages +describe the observed means, not statistical significance. Decreases are green and increases +are red when colors are enabled; signs remain visible without color. A percentage change from +a zero reference is undefined and displays as `N/A`. + +The **first metric is primary**: it is emphasized in terminal output and is the only metric +in non-JSON exports. All comparisons use the first command as their reference, regardless +of which metric is primary. There is no sorting or separate reference selection. + +| Metrics | Explicit units | +|:---|:---| +| `time_wall_clock`, `time_user`, `time_system` | `ns`, `us`, `ms`, `s`, `min`, `h` | +| `memory_peak_resident` | `B`, `kB`, `MB`, `GB`, `TB`, `KiB`, `MiB`, `GiB`, `TiB` | +| `cpu_cycles`, `instructions`, `cache_references`, `cache_misses`, `branch_misses` | `count`, `k`, `M`, `B` | + +Counter suffixes use decimal scaling: `k` is a thousand, `M` a million, and `B` a billion. +For example, `instructions:B` can display `143.2 B`. The metric determines the meaning of +`B`: billion for counters, bytes for memory. + +When a unit is omitted, terminal and markup output choose a suitable scale automatically. +CSV instead uses the base unit unless a unit is explicitly selected. JSON always uses base +units. Duplicate metrics, invalid units, and unavailable explicitly selected metrics are errors. +See [JSON](#json) below for metric definitions and platform availability. + +For peak memory, each sample is one run's peak; the displayed mean is the average of those +peaks, and the maximum is the largest observed peak. Automatic run counts still use elapsed +time to estimate the benchmark effort, regardless of the selected metrics. + ### Warmup runs and preparation commands For programs that perform a lot of disk I/O, the benchmarking results can be heavily influenced @@ -58,7 +107,7 @@ hyperfine -S --warmup 3 'grep -R TODO *' ``` Conversely, if you want to run the benchmark for a cold cache, you can use the `-p`/`--prepare` -option to run a special command before *each* timing run. For example, to clear Linux filesystem caches, +option to run a special command before *each* benchmark run. For example, to clear Linux filesystem caches, you can run ```sh sync; echo 3 | sudo tee /proc/sys/vm/drop_caches @@ -90,7 +139,7 @@ hyperfine -L compiler g++,clang++ '{compiler} -O2 main.cpp' ``` A common use case is comparing the same command across multiple Git branches. Use `--setup` -to switch branches once before each set of timing runs, so the branch switch is not part of +to switch branches once before each set of benchmark runs, so the branch switch is not part of the measured command: ```sh hyperfine \ @@ -129,7 +178,7 @@ If any of these commands need shell syntax, enable a shell explicitly. When a shell is enabled, hyperfine *corrects for the shell spawning time*. It runs the shell with an empty command multiple times to measure its startup time, then subtracts this time from each -measurement. +time measurement. Memory and counter measurements are not calibrated this way. ### Shell functions @@ -150,22 +199,35 @@ hyperfine -S 'my_function() { sleep 1; }; my_function' ### Exporting results -Hyperfine has multiple options for exporting benchmark results to CSV, JSON, Markdown and other -formats (see `--help` text for details). +Hyperfine can export results to CSV, JSON, Markdown, AsciiDoc, and org-mode. All exports +preserve input order. Non-JSON formats contain only the primary metric. + +CSV includes the metric and unit explicitly. An explicit unit in `--metrics` also applies +to CSV; otherwise, CSV uses seconds, bytes, or unscaled counts, without automatic scaling. +For example, `--metrics memory_peak_resident:MiB` exports memory in MiB, while +`--metrics memory_peak_resident` exports bytes. #### Markdown -You can use the `--export-markdown ` option to create tables like the following: +Markdown, AsciiDoc, and org-mode tables use the primary metric's selected unit, or choose +a suitable unit automatically when it is omitted. For example: -| Command | Mean [s] | Min [s] | Max [s] | Relative | -|:---|---:|---:|---:|---:| -| `find . -iregex '.*[0-9]\.jpg$'` | 2.275 ± 0.046 | 2.243 | 2.397 | 9.79 ± 0.22 | -| `find . -iname '*[0-9].jpg'` | 1.427 ± 0.026 | 1.405 | 1.468 | 6.14 ± 0.13 | -| `fd -HI '.*[0-9]\.jpg$'` | 0.232 ± 0.002 | 0.230 | 0.236 | 1.00 | +```sh +hyperfine --metrics memory_peak_resident:MiB,time_wall_clock:ms \ + --export-markdown results.md './baseline' './candidate' +``` + +The table contains peak-RSS statistics and comparisons against `./baseline`. Wall-clock time +still appears in the terminal and JSON, but not in this table. #### JSON -The JSON export includes the following metrics for each measured run (excluding warmup runs): +JSON records the primary metric and includes all collected metrics, regardless of the display +selection. Values use base units (seconds, bytes, and unscaled counts), independently of +any units selected with `--metrics`. Individual samples exclude warmup runs. +Results preserve input order; the first result is the reference. + +The metrics are: - **`time_wall_clock`**: Time from start to finish, including time spent waiting, in seconds. @@ -213,7 +275,9 @@ The JSON export includes the following metrics for each measured run (excluding - Linux: Same scope as `cpu_cycles`. - macOS/Windows: Currently not supported. -Hardware counters are not available if a `--shell` is used. +Hardware counters are not available when a shell is enabled. Availability also depends on the +platform and permissions. An unavailable metric can be omitted from JSON when it was not +explicitly selected; naming it in a `--metrics` list requires it to be collected. The JSON output is useful if you want to analyze the benchmark results in more detail. The [`scripts/`](https://github.com/sharkdp/hyperfine/tree/master/scripts) folder includes a lot @@ -227,7 +291,7 @@ multiple benchmarks: ### Detailed benchmark flowchart -The following chart explains the execution order of various timing runs when using options +The following chart explains the execution order of various benchmark runs when using options like `--warmup`, `--prepare `, `--setup ` or `--cleanup `: ![](doc/execution-order.png) diff --git a/doc/hyperfine.1 b/doc/hyperfine.1 index e986238..e0fbc18 100644 --- a/doc/hyperfine.1 +++ b/doc/hyperfine.1 @@ -15,10 +15,6 @@ hyperfine \- command\-line benchmarking tool .IR NUM ] .RB [ \-\-setup .IR CMD ] -.RB [ \-\-reference -.IR CMD ] -.RB [ \-\-reference\-name -.IR NAME ] .RB [ \-\-prepare .IR CMD ] .RB [ \-\-conclude @@ -38,10 +34,8 @@ hyperfine \- command\-line benchmarking tool .IR SHELL ] .RB [ \-\-style .IR TYPE ] -.RB [ \-\-sort -.IR METHOD ] -.RB [ \-\-time-unit -.IR UNIT ] +.RB [ \-\-metrics +.IR METRIC[:UNIT],...|default|all ] .RB [ \-\-export\-asciidoc .IR FILE ] .RB [ \-\-export\-csv @@ -75,7 +69,7 @@ A command\-line benchmarking tool which includes: * Warmup runs can be executed before the actual benchmark .RE .RS -* Cache-clearing commands can be set up before each timing run +* Cache-clearing commands can be set up before each benchmark run .RE .RS * Statistical outlier detection to detect interference from other programs and caching effects @@ -86,6 +80,16 @@ A command\-line benchmarking tool which includes: .RS * Parameterized benchmarks (e.g. vary the number of threads) .RE +.LP +Commands run and appear in input order. The first benchmark after parameter +expansion is always the reference for comparisons. Each command's results are +printed when it finishes, with one row per selected metric showing mean, +standard deviation, and range. Standard deviation is unavailable for a +single run. +Subsequent commands show the percentage change of their mean from the reference. +A positive change means more; a negative change means less. These changes do not +indicate statistical significance. A zero reference produces an undefined change +(displayed as N/A). All exports preserve input order. .SH OPTIONS .HP \fB\-w\fR, \fB\-\-warmup\fR \fINUM\fP @@ -109,28 +113,14 @@ Perform exactly \fINUM\fP runs for each command. If this option is not specified .HP \fB\-s\fR, \fB\-\-setup\fR \fICMD...\fP .IP -Execute \fICMD\fP once before each set of timing runs. This is useful +Execute \fICMD\fP once before each set of benchmark runs. This is useful for compiling your software with the provided parameters, or to do any other work that should happen once before a series of benchmark runs, not every time as would happen with the \fB\-\-prepare\fR option. .HP -\fB\-\-reference\fR \fICMD\fP -.IP -Use \fICMD\fP as the reference for the relative comparison of results. Without -parameters, benchmark \fICMD\fP as a separate reference command. If this option -is not specified, the fastest command is used as the reference. -With \fB\-\-parameter\-scan\fR or \fB\-\-parameter\-list\fR, select an existing -benchmark by its exact printed name. The name must match exactly one benchmark. -.HP -\fB\-\-reference\-name\fR \fINAME\fP -.IP -Give a meaningful \fINAME\fP to the reference command. -Requires \fB\-\-reference\fR. -Cannot be used with parameterized benchmarks; use \fB\-\-command\-name\fR instead. -.HP \fB\-p\fR, \fB\-\-prepare\fR \fICMD...\fP .IP -Execute \fICMD\fP before each timing run. This is useful for clearing disk caches, +Execute \fICMD\fP before each benchmark run. This is useful for clearing disk caches, for example. The \fB\-\-prepare\fR option can be specified once for all commands or multiple times, once for each command. In the latter case, each preparation command will be @@ -138,7 +128,7 @@ run prior to the corresponding benchmark command. .HP \fB\-C\fR, \fB\-\-conclude\fR \fICMD...\fP .IP -Execute \fICMD\fP after each timing run. This is useful for clearing disk caches, +Execute \fICMD\fP after each benchmark run. This is useful for clearing disk caches, for example. The \fB\-\-conclude\fR option can be specified once for all commands or multiple times, once for each command. In the latter case, each conclusion command will be @@ -248,65 +238,77 @@ interactive output without any colors. Set this to 'color' to keep the colors without any interactive output. Set this to 'none' to disable all the output of the tool. .HP -\fB\-\-sort\fR \fIMETHOD\fP +\fB\-\-metrics\fR \fIMETRIC[:UNIT],...|default|all\fP .IP -Specify the sort order of the speed comparison summary and the -exported tables for markup formats (Markdown, AsciiDoc, org\-mode): +Select the metrics to display, in order, with an optional unit for each. +Default: \fBtime_wall_clock,memory_peak_resident\fR, or \fBtime_wall_clock\fR +on Windows, where peak resident memory is not yet supported. +The first metric is primary: it is emphasized in terminal output and is the only +metric in non\-JSON exports. +The first benchmark has no change column; subsequent benchmarks show percentage +changes from the first benchmark. +.IP +Use \fBdefault\fR for the platform defaults or \fBall\fR for all available +metrics in the order listed below, with wall\-clock time primary. +Both presets select display units automatically and must be used alone, +without other metrics or unit suffixes. +The \fBall\fR preset skips metrics unsupported by the platform or shell, +or missing from any run of a benchmark. If a metric is unavailable for the +reference, its change displays as N/A. +.IP +Time metrics \fBtime_wall_clock\fR, \fBtime_user\fR, and \fBtime_system\fR +accept units \fBns\fR, \fBus\fR, \fBms\fR, \fBs\fR, \fBmin\fR, and \fBh\fR. +Peak memory, \fBmemory_peak_resident\fR, accepts \fBB\fR, \fBkB\fR, +\fBMB\fR, \fBGB\fR, \fBTB\fR, \fBKiB\fR, \fBMiB\fR, \fBGiB\fR, +and \fBTiB\fR. +.IP +Counter metrics \fBcpu_cycles\fR, \fBinstructions\fR, \fBcache_references\fR, +\fBcache_misses\fR, and \fBbranch_misses\fR accept \fBcount\fR (unscaled), +\fBk\fR (thousand), \fBM\fR (million), and \fBB\fR (billion). +For memory, \fBB\fR instead means bytes. +.IP +If a unit is omitted, terminal and markup output automatically select a suitable +unit. CSV uses base units unless a unit is explicitly selected. JSON always +uses base units, independently of this option. +.IP +Duplicate metrics, invalid units, and unavailable explicitly selected metrics are errors. +Hardware counters require direct execution (no shell) and platform support. +Automatic run counts continue to use elapsed time, regardless of the metrics. .RS -.IP "auto (default)" -the speed comparison will be ordered by time and -the markup tables will be ordered by command (input order). -.IP "command" -order benchmarks in the way they were specified -.IP "mean\-time" -order benchmarks by mean runtime -.RE -.HP -\fB\-u\fR, \fB\-\-time\-unit\fR \fIUNIT\fP -.IP -Set the time unit to be used. If the option is not given, the time unit is -determined automatically. Possible values: +Example: .RS -.IP "microsecond" -Aliases: µs, us, microseconds. -.IP "millisecond" -Aliases: ms, milliseconds. -.IP "second" -Aliases: s, seconds. -.IP "minute" -Aliases: min, minutes. -.IP "hour" -Aliases: h, hours. +\fBhyperfine\fR \fB\-\-metrics\fR memory_peak_resident:MiB,time_wall_clock:ms,instructions:B './baseline' './candidate' +.RE .RE -.IP -This option affects the standard output as well as all export formats -except for CSV and JSON. .HP \fB\-\-export\-asciidoc\fR \fIFILE\fP .IP -Export the timing summary statistics as an AsciiDoc table to the given \fIFILE\fP. -The output time unit can be changed using the \fB\-\-time\-unit\fR option. +Export the primary metric's summary statistics as an AsciiDoc table to the given +\fIFILE\fP. Use its selected unit, or choose a unit automatically when omitted. .HP \fB\-\-export\-csv\fR \fIFILE\fP .IP -Export the timing summary statistics as CSV to the given \fIFILE\fP. If you need the -timing results for each individual run, use the JSON export format. -The output time unit is always seconds. +Export the primary metric's summary statistics as CSV to the given \fIFILE\fP, +identifying the metric and unit explicitly. Use the explicitly selected unit, +or the base unit (seconds, bytes, or unscaled counts) when omitted. +CSV never scales units automatically. For individual samples, use JSON. .HP \fB\-\-export\-json\fR \fIFILE\fP .IP -Export the timing summary statistics and timings of individual runs as JSON to -the given \fIFILE\fP. The output time unit is always seconds. +Export summary statistics and individual samples for all collected metrics as +JSON to the given \fIFILE\fP. Include the primary metric, and always use base +units (seconds, bytes, or unscaled counts), regardless of display settings. .HP \fB\-\-export\-markdown\fR \fIFILE\fP .IP -Export the timing summary statistics as a Markdown table to the given \fIFILE\fP. -The output time unit can be changed using the \fB\-\-time\-unit\fR option. +Export the primary metric's summary statistics as a Markdown table to the given +\fIFILE\fP. Use its selected unit, or choose a unit automatically when omitted. .HP \fB\-\-export\-orgmode\fR \fIFILE\fP .IP -Export the timing summary statistics as an Emacs org\-mode table to the -given \fIFILE\fP. The output time unit can be changed using the \fB\-\-time\-unit\fR option. +Export the primary metric's summary statistics as an Emacs org\-mode table to +the given \fIFILE\fP. Use its selected unit, or choose a unit automatically +when omitted. .HP \fB\-\-show\-output\fR .IP diff --git a/src/benchmark/benchmark_result.rs b/src/benchmark/benchmark_result.rs index 1d9f597..35bd694 100644 --- a/src/benchmark/benchmark_result.rs +++ b/src/benchmark/benchmark_result.rs @@ -3,7 +3,6 @@ use std::collections::BTreeMap; use serde::Serialize; use crate::benchmark::measurement::Measurements; -use crate::quantity::Time; /// Parameter value #[derive(Debug, Default, Clone, Serialize, PartialEq)] @@ -47,9 +46,4 @@ impl BenchmarkResult { pub fn get_name(&self) -> &str { self.name.as_deref().unwrap_or(&self.command) } - - /// The average wall clock time - pub fn mean_wall_clock_time(&self) -> Time { - self.measurements.time_wall_clock_mean() - } } diff --git a/src/benchmark/measurement.rs b/src/benchmark/measurement.rs index d1ec75f..1877858 100644 --- a/src/benchmark/measurement.rs +++ b/src/benchmark/measurement.rs @@ -2,7 +2,7 @@ use std::process::ExitStatus; use serde::Serialize; -use crate::quantity::statistics::{max, mean, median, min, modified_zscores, standard_deviation}; +use crate::quantity::statistics::{mean, modified_zscores}; use crate::quantity::{serialize_information, serialize_time, Information, Time}; use crate::util::exit_code::extract_exit_code; @@ -131,32 +131,6 @@ impl Measurements { mean(self.wall_clock_times()) } - /// The standard deviation of all wall clock times. Not available if only one run has been performed - pub fn stddev(&self) -> Option