vsndrg/hyperfine
1
0 Fork 0
mirror of https://github.com/sharkdp/hyperfine.git synced 2026-10-28 08:08:58 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
master
hyperfine/CHANGELOG.md
T
David Peter b33755fe1f hyperfine 2.0
2026-10-07 23:06:41 +02:00

530 lines
22 KiB
Markdown
Raw Permalink Blame History

# v2.0.0
See the breaking changes below when migrating from hyperfine 1.x.
## Features
- Add support for alternative performance metrics (peak memory usage, instructions, CPU cycles, cache misses, branch misses, ...). By default, hyperfine will now display wall-clock time and peak memory usage, but users can use the [new `--metrics` option](README.md#choosing-metrics-and-units) to select different performance metrics. The terminal output now shows an overview of all selected performance metrics, including relative changes.
- Add `--env` to set environment variables even without an intermediate shell (which is the new default behavior, see below). For example:
```sh
# Parameterized benchmark, comparing different thread counts:
hyperfine -P threads 1 8 --env 'OMP_NUM_THREADS={threads}' 'my_command'
```
- The plotting and analysis scripts now support different metrics as well, see #981 and #984 (@sharkdp).
## Breaking changes
- Benchmarked commands are now executed directly by default, without an intermediate shell (`--shell=none`). Use `-S` (an alias for `--shell=default`) to restore the previous behavior (`sh` on Unix, `cmd.exe` on Windows), or select a shell with `--shell <SHELL>`. See [the rationale for this change](https://github.com/sharkdp/hyperfine/issues/787#issuecomment-6038898252).
- Always run `--setup`, `--prepare`, `--conclude`, and `--cleanup` commands in a shell, even when using `--shell=none` (the new default behavior), see #990 (@sharkdp).
- The JSON format for `--export-json` has changed (schema version 2). The [new format](README.md#json) now includes metadata, per-run measurements, and statistical summaries for all measured quantities, see #790 (@sharkdp). Code that reads those exported JSON files must be updated. For example, `results[i].mean` is now `results[i].summary.time_wall_clock.mean`, and `results[i].times` is replaced by `results[i].measurements[j].time_wall_clock.value`. The new structure looks like this:
```python
{
"schema_version": 2,
"primary_metric": "time_wall_clock",
"metadata": {
"hyperfine_version": "2.0.0",
"start_time": "2026-10-06T12:00:00Z",
"platform": {"os": "Linux", "architecture": "x86_64"}
},
"results": [
{
"command": "sleep 1", # Actual command after parameter substitution
"name": "wait 1s", # Optional custom or automatically generated name
"parameters": {"duration": {"value": "1"}}, # Values are now objects
"environment": {"OMP_NUM_THREADS": "8"}, # Only present with --env overrides
"measurements": [ # One entry per run, excluding warmups
{
"time_wall_clock": {"value": 1.0, "unit": "second"},
"time_cpu": {"value": 0.0, "unit": "second"},
"time_user": {"value": 0.0, "unit": "second"},
"time_system": {"value": 0.0, "unit": "second"},
"memory_peak_resident": {"value": 1048576.0, "unit": "byte"},
"cpu_cycles": {"value": 5000000}, # Hardware counters, when available
"instructions": {"value": 10000000},
"exit_code": 0
},
... # Further runs omitted
],
"summary": {
"time_wall_clock": {
"unit": "second",
"count": 2,
"mean": 1.0,
"stddev": 0.0,
"median": 1.0,
"min": 1.0,
"max": 1.0
}
# Other metrics have the same summary structure.
}
}
]
}
```
Times are always exported in seconds and memory in bytes. Hardware counters and their summaries have no `unit` field. Unavailable memory measurements and hardware counters are omitted; other available counters may also be included. `stddev` is `null` for a single run.
- `--time-unit`/`-u` has been removed. Units can now be selected using the `--metrics METRIC[:UNIT],…` option, e.g. `--metrics time_wall_clock:ms,memory_peak_resident:MiB`.
- `--sort` has been removed. It complicated hyperfine significantly and doesn't make too much sense with the new output format.
- `--reference` and `--reference-name` have been removed (for now). The first command is always considered as the reference command. Invocations using `--reference` now show migration guidance to put the reference command first.
- The format of [Markdown](README.md#markdown), AsciiDoc and org-mode, and [CSV](README.md#csv) exports has also been changed. Markdown, AsciiDoc, and org-mode contain only the primary metric (the first metric selected by `--metrics`). CSV exports all selected metrics side by side.
# v1.21.0
## Features
- Add support for minutes and hours in `--time-unit`, as well as short and plural aliases such as `ms`, `seconds`, and `min`, see #960 (@sharkdp)
- Expose `$HYPERFINE_ITERATION` to `--prepare` and `--conclude` commands, see #781 and #807 (@willcl-ark)
- Show elapsed time during the initial benchmark run, see #416 and #581 (@devonhollowood)
- Add colors to `--help` output, see #841 (@starsep)
## Changes
- With `--parameter-scan` or `--parameter-list`, `--reference` now selects an existing benchmark by its full displayed name instead of running a separate reference command. The name must match exactly one benchmark; use `--command-name` instead of `--reference-name` in this mode, see #847 and #979 (@sharkdp)
- Format times in CSV exports with six decimal places, see #966 and #972 (@sharkdp)
- Update dependencies and raise the minimum supported Rust version to 1.97, see #934 (@sharkdp)
## Bugfixes
- Make Markdown, AsciiDoc, and Org-mode exports compare against the specified reference, and label faster and slower results in command-sorted comparisons with `--reference`, see #811 and #979 (@sharkdp)
- Reject zero values for `--runs`, `--min-runs`, and `--max-runs` instead of allowing invalid run counts, see #923 (@VXNCXNX, @sharkdp)
- Reject negative parameter-scan steps before expanding ranges, avoiding hangs and excessive memory use, see #951 (@Likio3000)
- Preserve existing export files when command options are invalid, see #950 (@Likio3000)
- Truncate export files when rewriting results to avoid leaving trailing bytes, see #970 (@sharkdp)
- Preserve backticks in command names in Markdown exports, see #947 (@Likio3000)
- Handle broken output pipes gracefully instead of panicking, see #932 (@Mathjk)
- Collect peak memory usage separately for each command on Unix, so commands no longer inherit the peak memory usage of earlier commands, see #965 (@sharkdp)
- Preserve 100-nanosecond precision in Windows CPU times, see #964 (@sharkdp)
- Python scripts: Correct the interpretation of Welch's t-test results, see #943 (@sharkdp)
## Other
- Build binaries for Windows ARM64 and Linux ARM64 musl, see #922 and #924 (@meop)
- Refactor measurement, statistics, and formatting code to use typed quantities, see #954, #958, #961, #967, #969, #971, and #976 (@sharkdp)
- Improve documentation and examples, including comparisons across Git branches, iteration variables, reference commands, and selective failure handling, see #899, #900, #929, and #940 (@xfocus3, @ded-furby, @Likio3000, @sharkdp)
# v1.20.0
## Features
- Add `--reference-name` option to give a meaningful name to the reference command, see #808 (@niklasdewally)
- The `--ignore-failure` option now supports a comma-separated list of exit codes to ignore (e.g., `--ignore-failure=1,2`), see #836 (@sharkdp)
- Python scripts: Add `--time-unit` option to `advanced_statistics.py` (@sharkdp)
- Python scripts: Add new `plot_benchmark_comparison.py` script for plotting collections of benchmarks, see #806 (@marxin)
## Bugfixes
- Fix bug where naming individual commands with parameter scan was not working correctly, see #794 (@teofr)
## Other
- Restrict `cat` tests to Unix environments, see #776 and #777 (@ritvikos)
# v1.19.0
## Features
- Add a new `--reference <cmd>` option to specify a reference command for the relative speed comparison, see #579, #577 and #744 (@sharkdp)
- Add `--conclude` argument (analog to `--prepare`), see #565 and #719 (@jackoconnordev)
- Allow `--output=…` to appear once for each command, enabling use cases like `hyperfine --output=null my-cmd --output=./file.log my-cmd`, see #529 and #775 (@sharkdp)
- The environment variable `$HYPERFINE_ITERATION` will now contain the current iteration number for each benchmarked command, see #775 (@sharkdp)
- Add iteration information to failure error message, see #771 and #772 (@sharkdp)
- Python scripts:
- legend modification parameters and output DPI, see #758 (@Spreadcat)
- Nicer whiskers plot, see #727 (@serpent7776)
## Bugfixes
- ETA not clearly visible on terminals with a block cursor, see #698 and #699 (@overclockworked64)
- Fix zsh completions, see #717 (@xzfc)
## Other
- Build binaries for aarch64-apple-darwin, see #728 (@Phault)
- Various cleanups (@hamirmahal, @one230six)
# v1.18.0
## Features
- Add support for microseconds via `--time-unit microsecond`, see #684 (@sharkdp)
## Bugfixes
- Proper argument quoting on Windows CMD, see #296 and #678 (@PedroWitzel)
# v1.17.0
## Features
- Add new `--sort` option to control the order in the rel. speed comparison and in markup export formats, see #601, #614, #655 (@sharkdp)
- Parameters which are unused in the command line are now displayed in parentheses, see #600 and #644 (@sharkdp).
- Added `--log-count` option for histogram plots, see `scripts/plot_histogram.py` (@sharkdp)
## Changes
- Updated hyperfine to use `windows-sys` instead of the unmaintained `winapi`, see #624, #639, #636, #641 (@clemenswasser)
- Silenced deprecation warning in Python scripts, see #633 (@nicovank)
- Major update of the man page, see 0ce6578, #647 (@sharkdp)
## Bugfixes
- Do not export intermediate results to stdout when using `-` as a file name, see #640 and #643 (@sharkdp)
- Markup exporting does not fail if benchmark results are zero, see #642 (@sharkdp)
# v1.16.1
## Bugfixes
- Fix line-wrapping of `--help` text (@sharkdp)
- Fix `--input=null` (@sharkdp)
# v1.16.0
## Features
- Added new `--input` option, see #541 and #563 (@snease)
- Added possibility to specify `-` as the filename in the
`--export-*` options, see #615 and #623 (@humblepenguinn)
## Changes
- Improve hints for outlier warnings if `--warmup` or `--prepare` are in use already,
see #570 (@sharkdp)
## Bugfixes
- Fix uncolored output on Windows if `TERM` is not set, see #583 (@nabijaczleweli)
- On Windows, only run `cmd.exe` with the `/C` option. Use `-c` for all other shells.
See #568 and #582 (@FilipAndersson245)
## Other
- Thanks to @berombau for working on dependency upgrades, see #584
- Fixed installation on Windows, see #595 and #596 (@AntoniosBarotsis)
# v1.15.0
## Features
- Disable colorized output in case of `TERM=dumb` or `NO_COLOR=1`, see #542 and #555 (@nabijaczleweli)
- Add new (experimental) `--min-benchmarking-time <secs>` option, see #527 (@sharkdp)
## Bugfixes
- Fix user and kernel times on Windows, see #368 and #538 (@clemenswasser)
## Other
- Improve `--help` texts of `--export-*` options, see #506 and #522 (@Engineer-of-Efficiency)
# v1.14.0
## Features
- Add a new `--output={null,pipe,inherit,<FILE>}` option to control
where the output of the benchmarked program is redirected (if at all),
see #377 and #509 (@tavianator, originally suggested by @BurntSushi)
- Add Emacs org-mode as a new export format, see #491 (@ppaulweber)
# v1.13.0
## Features
- Added a new `--shell=none`/`-N` option to disable the intermediate
shell for executing the benchmarked commands. Hyperfine normally
measures and subtracts the shell spawning time, but the intermediate
shell always introduces a certain level of measurement noise. Using
`--shell=none`/`-N` allows users to benchmark very fast commands
(with a runtime on the order of a few milliseconds). See #336, #429,
and #487 (@cipriancraciun and @sharkdp)
- Added `--setup`/`-s` option that can be used to run `make all` or
similar. It runs once per set of tests, like `--cleanup`/`-c` (@avar)
- Added new `plot_progression.py` script to debug background interference
effects.
## Changes
- Breaking change: the `-s` short option for `--style` is now used for
the new `--setup` option.
- The environment offset randomization is now also available on Windows,
see #484
## Other
- Improved documentation and test coverage, cleaned up code base for
future improvements.
# v1.12.0
## Features
- `--command-name` can now take parameter names from `--parameter-*` options, see #351 and #391 (@silathdiir)
- Exit codes (or signals) are now printed in cases of command failures, see #342 (@KaindlJulian)
- Exit codes are now part of the JSON output, see #371 (@JordiChauzi)
- Colorized output should now be enabled on Windows by default, see #427
## Changes
- When `--export-*` commands are used, result files are created before benchmark execution
to fail early in case of, e.g., wrong permissions. See #306 (@s1ck).
- When `--export-*` options are used, result files are written after each individual
benchmark command instead of writing after all benchmarks have finished. See #306 (@s1ck).
- Reduce number of shell startup time measurements from 200 to 50, generally speeding up benchmarks. See #378
- User and system time are now in consistent time units, see #408 and #409 (@film42)
# v1.11.0
## Features
- The `-L`/`--parameter-list` option can now be specified multiple times to
evaluate all possible combinations of the listed parameters:
``` bash
hyperfine -L number 1,2 -L letter a,b,c \
"echo {number}{letter}" \
"printf '%s\n' {number}{letter}"
# runs 12 benchmarks: 2 commands (echo and printf) times 6 combinations of
# the "letter" and "number" parameters
```
See: #253, #318 (@wchargin)
- Add CLI option to identify a command with a custom name, see #326 (@scampi)
## Changes
- When parameters are used with `--parameter-list` or `--parameter-scan`, the JSON export format
now contains a dictionary `parameters` instead of a single key `parameter`. See #253, #318.
- The `plot_parametrized.py` script now infers the parameter name, and its `--parameter-name`
argument has been deprecated. See #253, #318.
## Bugfixes
- Fix a bug in the outlier detection which would only detect "slow outliers" but not the fast
ones (runs that are much faster than the rest of the benchmarking runs), see #329
- Better error messages for very fast commands that would lead to inf/nan results in the relative
speed comparison, see #319
- Show error message if `--warmup` or `--*runs` arguments can not be parsed, see #337
- Keep output colorized when the output is not interactive and `--style=full` or `--style=color` is used.
# v1.10.0
## Features
- Hyperfine now comes with shell completion files for Bash, Zsh, Fish
and PowerShell, see #290 (@four0000four).
- Hyperfine now comes with a basic man page, see #257 (@cadeef)
- During execution of benchmarks, hyperfine will now set a `HYPERFINE_RANDOMIZED_ENVIRONMENT_OFFSET` environment variable in order to randomize the memory layout. See #235 and #241 for references and details.
- A few enhancements for the histogram plotting scripts and the
advanced statistics script
- Updates for the `plot_whisker.py` script, see #275 (@ghaiklor)
## Bugfixes
- Fix Spin Icon on Windows, see #229
- A few typos have been fixed, see #292 (@McMartin)
## Packaging
- `hyperfine` is now available on MacPorts for macOS, see #281 (@herbygillot)
- `hyperfine` is now available on OpenBSD, see #289 (@minusf)
Package authors: note that Hyperfine now comes with a set of shell completion files and a man page (see above)
# v1.9.0
## Features
- The new `--parameter-list <VAR> <VALUES>` option can be used to run
a parametrized benchmark on a user-specified list of values.
This is similar to `--parameter-scan <VAR> <MIN> <MAX>`, but doesn't
necessarily require numeric arguments.
``` bash
hyperfine --parameter-list compiler "g++,clang++" \
"{compiler} -O2 main.cpp"
```
See: #227, #234 (@JuanPotato)
- Added `none` as a possible choice for the `--style` option to
run `hyperfine` without any output, see #193 (@knidarkness)
- Added a few new scripts for plotting various types of benchmark
results (https://github.com/sharkdp/hyperfine/tree/master/scripts)
## Changes
- The `--prepare` command is now also run during the warmup
phase, see #182 (@sseemayer)
- Better estimation of the remaining benchmark time due to an update
of the `indicatif` crate.
## Other
- `hyperfine` is now available on NixOS, see #240 (@tuxinaut)
# v1.8.0
## Features
- The `--prepare <CMD>` option can now be specified multiple times to
run specific preparation commands for each of the benchmarked programs:
``` bash
hyperfine --prepare "make clean; git checkout master" "make" \
--prepare "make clean; git checkout feature" "make"
```
See: #216, #218 (@iamsauravsharma)
- Added a new [`welch_ttest.py`](https://github.com/sharkdp/hyperfine/blob/master/scripts/welch_ttest.py) script to test whether or not the two benchmark
results are the same, see #222 (@uetchy)
- The Markdown export has been improved. The relative speed is now exported
with a higher precision (see #208) and includes the standard deviation
(see #225).
## Other
- Improved documentation for [`scripts`](https://github.com/sharkdp/hyperfine/tree/master/scripts) folder (@matthieusb)
# v1.7.0
## Features
- Added a new `-D`,`--parameter-step-size` option that can be used to control
the step size for `--parameter-scan` benchmarks. In addition, decimal numbers
are now allowed for parameter scans. For example, the following command runs
`sleep 0.3`, `sleep 0.5` and `sleep 0.7`:
``` bash
hyperfine --parameter-scan delay 0.3 0.7 -D 0.2 'sleep {delay}'
```
For more details, see #184 (@piyushrungta25)
## Other
- hyperfine is now in the official Alpine repositories, see #177 (@maxice8, @5paceToast)
- hyperfine is now in the official Fedora repositories, see #196 (@ignatenkobrain)
- hyperfine is now in the official Arch Linux repositories
- hyperfine can be installed on FreeBSD, see #204 (@0mp)
- Enabled LTO for slightly smaller binary sizes, see #179 (@Calinou)
- Various small improvements all over the code base, see #194 (@phimuemue)
# v1.6.0
## Features
- Added a `-c, --cleanup <CMD>` option to execute `CMD` after the completion of all benchmarking runs for a given command. This is useful if the commands to be benchmarked produce artifacts that need to be cleaned up. See #91 (@RalfJung and @colinwahl)
- Add parameter values (for `--parameter-scan` benchmarks) to exported CSV and JSON files. See #131 (@bbannier)
- Added AsciiDoc export option, see #137 (@5paceToast)
- The relative speed is now part of the Markdown export, see #127 (@mathiasrw and @sharkdp).
- The *median* run time is now exported via CSV and JSON, see #171 (@hosewiejacke and @sharkdp).
## Other
- Hyperfine has been updated to Rust 2018 (@AnderEnder). The minimum supported Rust version is now 1.31.
# v1.5.0
## Features
- Show the number of runs in `hyperfine`s output (@tcmal)
- Added two Python scripts to post-process exported benchmark results (see [`scripts/`](https://github.com/sharkdp/hyperfine/tree/master/scripts) folder)
## Other
- Refined `--help` text for the `--export-*` flags (@psteinb)
- Added Snapcraft file (@popey)
- Small improvements in the progress bar "experience".
# v1.4.0
## Features
- Added `-S`/`--shell` option to override the default shell, see #61 (@mqudsi and @jasonpeacock)
- Added `-u`/`--time-unit` option to change the unit of time (`second` or `millisecond`), see #80 (@jasonpeacock)
- Markdown export auto-selects time unit, see #71 (@jasonpeacock)
# v1.3.0
## Feature
- Compute and print standard deviation of the speed ratio, see #83 (@Shnatsel)
- More compact output format, see #70 (@jasonpeacock)
- Added `--style=color`, see #70 (@jasonpeacock)
- Added options to specify the max/exact numbers of runs, see #77 (@orium)
## Bugfixes
- Change Windows `cmd` interpreter to `cmd.exe` to prevent accidentally calling other programs, see #74 (@tathanhdinh)
## Other
- Binary releases for Windows are now available, see #87
# v1.2.0
- Support parameters in preparation commands, see #68 (@siiptuo)
- Updated dependencies, see #69. The minimum required Rust version is now 1.24.
# v1.1.0
* Added `--show-output` option (@chrisduerr and @sevagh)
* Refactoring work (@stevepentland)
# v1.0.0
## Features
* Support for various export-formats like CSV, JSON and Markdown - see #38, #44, #49, #42 (@stevepentland)
* Summary output that compares the different benchmarks, see #6 (@stevepentland)
* Parameterized benchmarks via `-P`, `--parameter-scan <VAR> <MIN> <MAX>`, see #19
## Thanks
I'd like to say a big THANK YOU to @stevepentland for implementing new features,
for reviewing pull requests and for giving very valuable feedback.
# v0.5.0
* Proper Windows support (@stevepentland)
* Added `--style auto/basic/nocolor/full` option (@stevepentland)
* Correctly estimate the full execution time, see #27 (@rleungx)
* Added Void Linux install instructions (@wpbirney)
# v0.4.0
- New `--style` option to disable output coloring and interactive CLI features, see #24 (@stevepentland)
- Statistical outlier detection, see #23 #18
# v0.3.0
## Features
- In addition to 'real' (wall clock) time, Hyperfine can now also measure 'user' and 'system' time (see #5).
- Added `--prepare` option that can be used to clear up disk caches before timing runs, for example (see #8).
## Other
- [Arch Linux package](https://aur.archlinux.org/packages/hyperfine) for Hyperfine (@jD91mZM2).
- Ubuntu/Debian packages are now available.
# v0.2.0
Initial public release
Reference in new issue View Git Blame Copy Permalink