diff --git a/CHANGELOG.md b/CHANGELOG.md index 375894f..2ed7e7b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ ## Breaking changes -- Replace `--time-unit`/`-u` with `--metrics METRIC[:UNIT],…`. Select multiple metrics for terminal output; the first metric is primary for all non-JSON exports. Presets include `default`, `speed` (wall-clock, user, and system time), and `all` (all available metrics). Units are chosen automatically for terminal and markup output when omitted. Counters support decimal `k`, `M`, and `B` suffixes. +- Replace `--time-unit`/`-u` with `--metrics METRIC[:UNIT],…`. Select multiple metrics for terminal output; the first metric is primary for all non-JSON exports. Presets include `default`, `time` (wall-clock, total CPU, user, and system time), and `all` (all available metrics). The `time_cpu` metric sums user and system time for each run. Units are chosen automatically for terminal and markup output when omitted. Counters support decimal `k`, `M`, and `B` suffixes. - Remove `--sort`, `--reference`, and `--reference-name`. Commands and exports always remain in input order, and the first expanded benchmark is the reference for every metric. - Replace time-only terminal output and the final speed ranking with a per-command metric table. Show wall-clock time and peak resident memory by default (wall-clock time only on Windows), with percentage changes from the first command for subsequent benchmarks. Color all metric rows and make the entire primary row bold. - Export only the primary metric to CSV, Markdown, AsciiDoc, and org-mode. CSV uses an explicitly selected unit or the metric's base unit when omitted. JSON continues to contain all collected metrics in base units and identifies the primary metric. diff --git a/README.md b/README.md index 734df42..29b49bc 100644 --- a/README.md +++ b/README.md @@ -58,9 +58,9 @@ hyperfine \ './baseline' './candidate' ``` -Use `--metrics=default` for the platform defaults, `--metrics=speed` for -`time_wall_clock,time_user,time_system`, or `--metrics=all` for all available metrics in the -order listed below. The `speed` preset works on all platforms, including with a shell. +Use `--metrics=default` for the platform defaults, `--metrics=time` for +`time_wall_clock,time_cpu,time_user,time_system`, or `--metrics=all` for all available metrics in the +order listed below. The `time` preset works on all platforms, including with a shell. All presets keep wall-clock time primary, choose display units automatically, and must be used alone, without other metrics or unit suffixes. The `all` preset skips metrics unsupported by the platform or shell, or missing from any run @@ -81,7 +81,7 @@ separate reference selection. | Metrics | Explicit units | |:---|:---| -| `time_wall_clock`, `time_user`, `time_system` | `ns`, `us`, `ms`, `s`, `min`, `h` | +| `time_wall_clock`, `time_cpu`, `time_user`, `time_system` | `ns`, `us`, `ms`, `s`, `min`, `h` | | `memory_peak_resident` | `B`, `kB`, `MB`, `GB`, `TB`, `KiB`, `MiB`, `GiB`, `TiB` | | `cpu_cycles`, `instructions`, `cache_references`, `cache_misses`, `branch_misses` | `count`, `k`, `M`, `B` | @@ -234,17 +234,23 @@ The metrics are: - **`time_wall_clock`**: Time from start to finish, including time spent waiting, in seconds. -- **`time_user`**: CPU time spent running the program's code, summed across threads, in seconds. +- **`time_user`**: CPU time spent executing application and library code in user mode, + summed across threads, in seconds. - Linux/macOS: Includes child-process time when parents wait for their children to finish. - Windows: Includes the command and its child processes. -- **`time_system`**: CPU time spent running operating-system code for the program, for example +- **`time_system`**: CPU time spent executing kernel code on the program's behalf, for example to read files, summed across threads, in seconds. - Linux/macOS: Includes child-process time when parents wait for their children to finish. - Windows: Includes the command and its child processes. +- **`time_cpu`**: Total CPU time, calculated as `time_user + time_system` for each run, in seconds. + It excludes time spent sleeping or waiting without executing. CPU time is summed across + threads, so it can exceed wall-clock time: four threads running on four cores for one second + can consume roughly four CPU-seconds. + - **`memory_peak_resident`**: Peak memory held in physical RAM, in bytes. - Linux/macOS: Peak resident set size (RSS). This is the largest per-process peak among the diff --git a/doc/hyperfine.1 b/doc/hyperfine.1 index 8f743fc..eded46e 100644 --- a/doc/hyperfine.1 +++ b/doc/hyperfine.1 @@ -35,7 +35,7 @@ hyperfine \- command\-line benchmarking tool .RB [ \-\-style .IR TYPE ] .RB [ \-\-metrics -.IR METRIC[:UNIT],...|default|speed|all ] +.IR METRIC[:UNIT],...|default|time|all ] .RB [ \-\-export\-asciidoc .IR FILE ] .RB [ \-\-export\-csv @@ -238,7 +238,7 @@ interactive output without any colors. Set this to 'color' to keep the colors without any interactive output. Set this to 'none' to disable all the output of the tool. .HP -\fB\-\-metrics\fR \fIMETRIC[:UNIT],...|default|speed|all\fP +\fB\-\-metrics\fR \fIMETRIC[:UNIT],...|default|time|all\fP .IP Select the metrics to display, in order, with an optional unit for each. Default: \fBtime_wall_clock,memory_peak_resident\fR, or \fBtime_wall_clock\fR @@ -249,9 +249,9 @@ The first benchmark has no change column; subsequent benchmarks show percentage changes from the first benchmark. Decreases are green and increases are red when colors are enabled. .IP -Use \fBdefault\fR for the platform defaults, \fBspeed\fR for -\fBtime_wall_clock,time_user,time_system\fR, or \fBall\fR for all available -metrics in the order listed below. The \fBspeed\fR preset works on all +Use \fBdefault\fR for the platform defaults, \fBtime\fR for +\fBtime_wall_clock,time_cpu,time_user,time_system\fR, or \fBall\fR for all available +metrics in the order listed below. The \fBtime\fR preset works on all platforms, including with a shell. All presets keep wall\-clock time primary, select display units automatically, and must be used alone, without other metrics or unit suffixes. @@ -259,8 +259,10 @@ The \fBall\fR preset skips metrics unsupported by the platform or shell, or missing from any run of a benchmark. If a metric is unavailable for the reference, its change displays as N/A. .IP -Time metrics \fBtime_wall_clock\fR, \fBtime_user\fR, and \fBtime_system\fR +Time metrics \fBtime_wall_clock\fR, \fBtime_cpu\fR, \fBtime_user\fR, and \fBtime_system\fR accept units \fBns\fR, \fBus\fR, \fBms\fR, \fBs\fR, \fBmin\fR, and \fBh\fR. +Total CPU time, \fBtime_cpu\fR, is the sum of user and system time for each run. +It is summed across threads and can exceed wall\-clock time when work runs in parallel. Peak memory, \fBmemory_peak_resident\fR, accepts \fBB\fR, \fBkB\fR, \fBMB\fR, \fBGB\fR, \fBTB\fR, \fBKiB\fR, \fBMiB\fR, \fBGiB\fR, and \fBTiB\fR. diff --git a/scripts/README.md b/scripts/README.md index 0b635b4..03c0ad5 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -7,14 +7,32 @@ hyperfine 'sleep 0.020' 'sleep 0.021' 'sleep 0.022' --export-json sleep.json uv run plot_whisker.py sleep.json ``` -All plotting scripts accept `--metric` to select a metric. Hardware counters -(`cpu_cycles`, `instructions`, `cache_references`, `cache_misses`, and `branch_misses`) -are plotted as unscaled counts when available in the JSON export. For example: +All plotting and analysis scripts use the JSON export's `primary_metric` by default. +Pass `--metric` to select a different metric. Supported metrics are `time_wall_clock`, +`time_cpu`, `time_user`, `time_system`, `memory_peak_resident`, `cpu_cycles`, +`instructions`, `cache_references`, `cache_misses`, and `branch_misses`, provided the +selected metric is available in every benchmark being analyzed. For example: ```bash uv run plot_histogram.py benchmark.json --metric instructions ``` +Time is displayed in seconds, memory in MiB, and counters as unscaled counts. +Multi-file plots require matching primary metrics unless `--metric` explicitly selects +one metric across all files. Exports without `primary_metric` fall back to wall-clock +time; the per-run `measurements` JSON schema is still required. + +For descriptive statistics or a Welch's t-test comparing exactly two benchmark results: + +```bash +uv run advanced_statistics.py benchmark.json --metric time_cpu --time-unit millisecond +uv run welch_ttest.py comparison.json --metric memory_peak_resident +``` + +`advanced_statistics.py` accepts `--time-unit second` or `--time-unit millisecond` +for time metrics only. Both analysis scripts also accept the default primary metric +without `--metric`. + ### Prerequisites Install [`uv`](https://docs.astral.sh/uv/getting-started/installation/) and run the diff --git a/scripts/advanced_statistics.py b/scripts/advanced_statistics.py index 99394eb..1a8ddda 100755 --- a/scripts/advanced_statistics.py +++ b/scripts/advanced_statistics.py @@ -7,74 +7,56 @@ # /// import argparse -import json -from enum import Enum import numpy as np - -class Unit(Enum): - SECOND = 1 - MILLISECOND = 2 - - def factor(self): - match self: - case Unit.SECOND: - return 1 - case Unit.MILLISECOND: - return 1e3 - - def __str__(self): - match self: - case Unit.SECOND: - return "s" - case Unit.MILLISECOND: - return "ms" - +from plot_utils import METRICS, add_metric_argument, load_results parser = argparse.ArgumentParser() parser.add_argument("file", help="JSON file with benchmark results") +add_metric_argument(parser) parser.add_argument( "--time-unit", - help="The unit of time.", - default="second", - action="store", + help="Display unit for time metrics (default: second).", choices=["second", "millisecond"], - dest="unit", ) args = parser.parse_args() -unit = Unit.MILLISECOND if args.unit == "millisecond" else Unit.SECOND -unit_str = str(unit) +metric, [results] = load_results(parser, [args.file], args.metric) +description = METRICS[metric] +unit, scale = description.unit, description.scale +if args.time_unit is not None: + if not metric.startswith("time_"): + parser.error("--time-unit can only be used with a time metric") + if args.time_unit == "millisecond": + unit, scale = "ms", 0.001 -with open(args.file) as f: - results = json.load(f)["results"] +def format_value(value): + suffix = f" {unit}" if unit else "" + return f"{value:.3f}{suffix}" + + +print(f"Metric: {description.label}\n") for result in results: - summary = result["summary"]["time_wall_clock"] - ts = [m["time_wall_clock"]["value"] * unit.factor() for m in result["measurements"]] + summary = result["summary"][metric] + values = [m[metric]["value"] / scale for m in result["measurements"]] - p05 = np.percentile(ts, 5) - p25 = np.percentile(ts, 25) - p75 = np.percentile(ts, 75) - p95 = np.percentile(ts, 95) + p05, p25, p75, p95 = np.percentile(values, [5, 25, 75, 95]) iqr = p75 - p25 print(f"Command '{result.get('name', result['command'])}'") print(f" runs: {summary['count']:8d}") - print(f" mean: {summary['mean'] * unit.factor():8.3f} {unit_str}") - if summary["stddev"] is None: - print(f" stddev: {'N/A':>8}") - else: - print(f" stddev: {summary['stddev'] * unit.factor():8.3f} {unit_str}") - print(f" median: {summary['median'] * unit.factor():8.3f} {unit_str}") - print(f" min: {summary['min'] * unit.factor():8.3f} {unit_str}") - print(f" max: {summary['max'] * unit.factor():8.3f} {unit_str}") + for statistic in ["mean", "stddev", "median", "min", "max"]: + value = summary[statistic] + formatted = "N/A" if value is None else format_value(value / scale) + print(f" {statistic + ':':<8}{formatted:>8}") print() print(" percentiles:") - print(f" P_05 .. P_95: {p05:.3f} {unit_str} .. {p95:.3f} {unit_str}") + print(f" P_05 .. P_95: {format_value(p05)} .. {format_value(p95)}") print( - f" P_25 .. P_75: {p25:.3f} {unit_str} .. {p75:.3f} {unit_str} (IQR = {iqr:.3f} {unit_str})" + f" P_25 .. P_75: {format_value(p25)} .. {format_value(p75)}" + f" (IQR = {format_value(iqr)})" ) print() diff --git a/scripts/plot_benchmark_comparison.py b/scripts/plot_benchmark_comparison.py index 44f66d9..a0e7459 100755 --- a/scripts/plot_benchmark_comparison.py +++ b/scripts/plot_benchmark_comparison.py @@ -14,13 +14,12 @@ Note all the input files must contain results for all commands. """ import argparse -import json import pathlib import matplotlib.pyplot as plt import numpy as np -from plot_utils import METRICS, add_metric_argument, validate_metric +from plot_utils import METRICS, add_metric_argument, load_results parser = argparse.ArgumentParser(description=__doc__) add_metric_argument(parser) @@ -34,7 +33,8 @@ parser.add_argument( parser.add_argument("-o", "--output", help="Save image to the given filename") args = parser.parse_args() -metric_label, metric_scale = METRICS[args.metric] +metric, datasets = load_results(parser, args.files, args.metric) +metric_info = METRICS[metric] commands = None data = [] @@ -45,10 +45,7 @@ if args.benchmark_names: "Number of benchmark names must match the number of input files." ) -for i, filename in enumerate(args.files): - with open(filename) as f: - results = json.load(f)["results"] - validate_metric(parser, results, args.metric) +for i, (filename, results) in enumerate(zip(args.files, datasets)): benchmark_commands = [b.get("name", b["command"]) for b in results] if commands is None: commands = benchmark_commands @@ -56,7 +53,7 @@ for i, filename in enumerate(args.files): assert commands == benchmark_commands, ( f"Unexpected commands in {filename}: {benchmark_commands}, expected: {commands}" ) - data.append([b["summary"][args.metric]["mean"] / metric_scale for b in results]) + data.append([b["summary"][metric]["mean"] / metric_info.scale for b in results]) if args.benchmark_names: inputs.append(args.benchmark_names[i]) else: @@ -79,7 +76,7 @@ ax.grid(visible=True, axis="y") if args.title: plt.title(args.title) plt.xlabel("Benchmark") -plt.ylabel(metric_label) +plt.ylabel(metric_info.axis_label) plt.legend(title="Command") if args.output: diff --git a/scripts/plot_histogram.py b/scripts/plot_histogram.py index c757ce2..46b15ce 100755 --- a/scripts/plot_histogram.py +++ b/scripts/plot_histogram.py @@ -11,12 +11,11 @@ """This program shows `hyperfine` benchmark results as a histogram.""" import argparse -import json import matplotlib.pyplot as plt import numpy as np -from plot_utils import METRICS, add_metric_argument, validate_metric +from plot_utils import METRICS, add_metric_argument, load_results parser = argparse.ArgumentParser(description=__doc__) add_metric_argument(parser) @@ -70,18 +69,15 @@ parser.add_argument( ) args = parser.parse_args() -metric_label, metric_scale = METRICS[args.metric] - -with open(args.file) as f: - results = json.load(f)["results"] -validate_metric(parser, results, args.metric) +metric, [results] = load_results(parser, [args.file], args.metric) +metric_info = METRICS[metric] if args.labels: labels = args.labels.split(",") else: labels = [b.get("name", b["command"]) for b in results] all_values = [ - [m[args.metric]["value"] / metric_scale for m in b["measurements"]] for b in results + [m[metric]["value"] / metric_info.scale for m in b["measurements"]] for b in results ] value_min = ( @@ -113,7 +109,7 @@ plt.legend( prop={"size": 10, "family": ["Source Code Pro", "Fira Mono", "Courier New"]}, ) -plt.xlabel(metric_label) +plt.xlabel(metric_info.axis_label) if args.title: plt.title(args.title) diff --git a/scripts/plot_parametrized.py b/scripts/plot_parametrized.py index 4973347..724f0b2 100755 --- a/scripts/plot_parametrized.py +++ b/scripts/plot_parametrized.py @@ -11,12 +11,11 @@ errorbar plot.""" import argparse -import json import sys import matplotlib.pyplot as plt -from plot_utils import METRICS, add_metric_argument, validate_metric +from plot_utils import METRICS, add_metric_argument, load_results parser = argparse.ArgumentParser(description=__doc__) add_metric_argument(parser) @@ -43,7 +42,8 @@ parser.add_argument( parser.add_argument("-o", "--output", help="Save image to the given filename.") args = parser.parse_args() -metric_label, metric_scale = METRICS[args.metric] +metric, datasets = load_results(parser, args.file, args.metric) +metric_info = METRICS[metric] if args.parameter_name is not None: sys.stderr.write( "warning: --parameter-name is deprecated; names are inferred from " @@ -84,11 +84,7 @@ def unique_parameter(benchmark): parameter_name = None -for filename in args.file: - with open(filename) as f: - results = json.load(f)["results"] - validate_metric(parser, results, args.metric) - +for results in datasets: (this_parameter_name, parameter_values) = extract_parameters(results) if parameter_name is not None and this_parameter_name != parameter_name: die( @@ -96,17 +92,17 @@ for filename in args.file: ) parameter_name = this_parameter_name - summaries = [b["summary"][args.metric] for b in results] - means = [s["mean"] / metric_scale for s in summaries] + summaries = [b["summary"][metric] for b in results] + means = [s["mean"] / metric_info.scale for s in summaries] stddevs = [ - s["stddev"] / metric_scale if s["stddev"] is not None else float("nan") + s["stddev"] / metric_info.scale if s["stddev"] is not None else float("nan") for s in summaries ] plt.errorbar(x=parameter_values, y=means, yerr=stddevs, capsize=2) plt.xlabel(parameter_name) -plt.ylabel(metric_label) +plt.ylabel(metric_info.axis_label) if args.log_y: plt.yscale("log") diff --git a/scripts/plot_progression.py b/scripts/plot_progression.py index e0edf67..c84c482 100755 --- a/scripts/plot_progression.py +++ b/scripts/plot_progression.py @@ -14,12 +14,11 @@ thermal throttling and similar effects. """ import argparse -import json import matplotlib.pyplot as plt import numpy as np -from plot_utils import METRICS, add_metric_argument, validate_metric +from plot_utils import METRICS, add_metric_argument, load_results def moving_average(values, num_runs): @@ -50,15 +49,12 @@ parser.add_argument( args = parser.parse_args() -metric_label, metric_scale = METRICS[args.metric] - -with open(args.file) as f: - results = json.load(f)["results"] -validate_metric(parser, results, args.metric) +metric, [results] = load_results(parser, [args.file], args.metric) +metric_info = METRICS[metric] for result in results: label = result.get("name", result["command"]) - values = [m[args.metric]["value"] / metric_scale for m in result["measurements"]] + values = [m[metric]["value"] / metric_info.scale for m in result["measurements"]] num = len(values) nums = range(num) @@ -86,7 +82,7 @@ for result in results: legend.append("moving average") plt.legend(legend) -plt.ylabel(metric_label) +plt.ylabel(metric_info.axis_label) if args.output: plt.savefig(args.output) diff --git a/scripts/plot_utils.py b/scripts/plot_utils.py index 842ded9..1eed2aa 100644 --- a/scripts/plot_utils.py +++ b/scripts/plot_utils.py @@ -1,15 +1,30 @@ -"""Shared metric options for the plotting scripts.""" +"""Shared metric selection and units for analysis and plotting scripts.""" + +import json +from typing import NamedTuple + + +class Metric(NamedTuple): + label: str + unit: str = "" + scale: float = 1 + + @property + def axis_label(self): + return f"{self.label} [{self.unit}]" if self.unit else self.label + METRICS = { - "time_wall_clock": ("Time [s]", 1), - "time_user": ("User CPU time [s]", 1), - "time_system": ("System CPU time [s]", 1), - "memory_peak_resident": ("Peak resident memory [MiB]", 1024**2), - "cpu_cycles": ("CPU cycles", 1), - "instructions": ("Completed CPU instructions", 1), - "cache_references": ("Cache references", 1), - "cache_misses": ("Cache misses", 1), - "branch_misses": ("Mispredicted branches", 1), + "time_wall_clock": Metric("Wall-clock time", "s"), + "time_cpu": Metric("Total CPU time", "s"), + "time_user": Metric("User CPU time", "s"), + "time_system": Metric("System CPU time", "s"), + "memory_peak_resident": Metric("Peak resident memory", "MiB", 1024**2), + "cpu_cycles": Metric("CPU cycles"), + "instructions": Metric("Completed CPU instructions"), + "cache_references": Metric("Cache references"), + "cache_misses": Metric("Cache misses"), + "branch_misses": Metric("Mispredicted branches"), } @@ -17,8 +32,8 @@ def add_metric_argument(parser): parser.add_argument( "--metric", choices=METRICS, - default="time_wall_clock", - help="Metric to plot (default: time_wall_clock). " + help="Metric to analyze (default: primary_metric from the JSON export, " + "or time_wall_clock if unspecified). " "Times are in seconds; memory is in MiB; hardware counters are unscaled counts.", ) @@ -30,3 +45,30 @@ def validate_metric(parser, results, metric): f"metric {metric!r} is unavailable for " f"{result.get('name', result['command'])!r}" ) + + +def load_results(parser, filenames, metric=None): + exports = [] + for filename in filenames: + with open(filename, encoding="utf-8") as f: + export = json.load(f) + if not export["results"]: + parser.error(f"no benchmark results in {filename}") + exports.append(export) + + if metric is None: + primary_metrics = { + export.get("primary_metric", "time_wall_clock") for export in exports + } + if len(primary_metrics) != 1: + parser.error( + "input files use different primary metrics; select one with --metric" + ) + metric = primary_metrics.pop() + if metric not in METRICS: + parser.error(f"unknown metric {metric!r}; select one with --metric") + + results = [export["results"] for export in exports] + for benchmarks in results: + validate_metric(parser, benchmarks, metric) + return metric, results diff --git a/scripts/plot_whisker.py b/scripts/plot_whisker.py index e53d375..d3c33b8 100755 --- a/scripts/plot_whisker.py +++ b/scripts/plot_whisker.py @@ -16,11 +16,10 @@ Quoting from the matplotlib documentation: """ import argparse -import json import matplotlib.pyplot as plt -from plot_utils import METRICS, add_metric_argument, validate_metric +from plot_utils import METRICS, add_metric_argument, load_results parser = argparse.ArgumentParser(description=__doc__) add_metric_argument(parser) @@ -33,22 +32,19 @@ parser.add_argument( parser.add_argument("-o", "--output", help="Save image to the given filename.") args = parser.parse_args() -metric_label, metric_scale = METRICS[args.metric] - -with open(args.file, encoding="utf-8") as f: - results = json.load(f)["results"] -validate_metric(parser, results, args.metric) +metric, [results] = load_results(parser, [args.file], args.metric) +metric_info = METRICS[metric] if args.labels: labels = args.labels.split(",") else: labels = [b.get("name", b["command"]) for b in results] values = [ - [m[args.metric]["value"] / metric_scale for m in b["measurements"]] for b in results + [m[metric]["value"] / metric_info.scale for m in b["measurements"]] for b in results ] if args.sort_by == "median": - medians = [b["summary"][args.metric]["median"] for b in results] + medians = [b["summary"][metric]["median"] for b in results] indices = sorted(range(len(labels)), key=lambda k: medians[k]) labels = [labels[i] for i in indices] values = [values[i] for i in indices] @@ -64,7 +60,7 @@ for patch, color in zip(boxplot["boxes"], colors): if args.title: plt.title(args.title) plt.legend(handles=boxplot["boxes"], labels=labels, loc="best", fontsize="medium") -plt.ylabel(metric_label) +plt.ylabel(metric_info.axis_label) plt.ylim(0, None) plt.xticks(list(range(1, len(labels) + 1)), labels, rotation=45) if args.output: diff --git a/scripts/welch_ttest.py b/scripts/welch_ttest.py index f6e48b3..1f76b37 100755 --- a/scripts/welch_ttest.py +++ b/scripts/welch_ttest.py @@ -7,32 +7,37 @@ # /// """This script performs Welch's t-test on a JSON export file with two -benchmark results to test for a difference in their population mean runtimes.""" +benchmark results to test for a difference in their population means.""" import argparse -import json -import sys +import math from scipy import stats +from plot_utils import METRICS, add_metric_argument, load_results + parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("file", help="JSON file with two benchmark results") +add_metric_argument(parser) args = parser.parse_args() -with open(args.file) as f: - results = json.load(f)["results"] +metric, [results] = load_results(parser, [args.file], args.metric) if len(results) != 2: - print("The input file has to contain exactly two benchmarks") - sys.exit(1) + parser.error("the input file must contain exactly two benchmarks") a, b = (x.get("name", x["command"]) for x in results[:2]) -X, Y = ([m["time_wall_clock"]["value"] for m in x["measurements"]] for x in results[:2]) +X, Y = ([m[metric]["value"] for m in x["measurements"]] for x in results[:2]) +if min(len(X), len(Y)) < 2: + parser.error("Welch's t-test requires at least two measurements per benchmark") +print(f"Metric: {METRICS[metric].label}") print(f"Command 1: {a}") print(f"Command 2: {b}\n") t, p = stats.ttest_ind(X, Y, equal_var=False) +if not math.isfinite(p): + parser.error("Welch's t-test is undefined for these measurements") th = 0.05 dispose = p < th print(f"t = {t:.3}, p = {p:.3}") @@ -40,9 +45,9 @@ print() if dispose: print( - f"A statistically significant difference in mean runtime was detected (p < {th})." + f"A statistically significant difference in mean value was detected (p < {th})." ) else: print( - f"No statistically significant difference in mean runtime was detected (p >= {th})." + f"No statistically significant difference in mean value was detected (p >= {th})." ) diff --git a/src/benchmark/measurement.rs b/src/benchmark/measurement.rs index 1877858..de19efc 100644 --- a/src/benchmark/measurement.rs +++ b/src/benchmark/measurement.rs @@ -100,8 +100,34 @@ pub struct Measurement { pub exit_status: ExitStatus, } +impl Measurement { + /// Total CPU time for this run, including user and kernel mode. + pub fn time_cpu(&self) -> Time { + self.time_user + self.time_system + } +} + +fn serialize_measurements(measurements: &[Measurement], serializer: S) -> Result +where + S: serde::Serializer, +{ + #[derive(Serialize)] + struct WithCpuTime<'a> { + #[serde(flatten)] + measurement: &'a Measurement, + #[serde(serialize_with = "serialize_time")] + time_cpu: Time, + } + + serializer.collect_seq(measurements.iter().map(|measurement| WithCpuTime { + measurement, + time_cpu: measurement.time_cpu(), + })) +} + #[derive(Debug, Default, Clone, Serialize, PartialEq)] pub struct Measurements { + #[serde(serialize_with = "serialize_measurements")] pub measurements: Vec, } diff --git a/src/benchmark/scheduler.rs b/src/benchmark/scheduler.rs index eb1ae81..c6431dc 100644 --- a/src/benchmark/scheduler.rs +++ b/src/benchmark/scheduler.rs @@ -81,7 +81,7 @@ fn generate_results(args: &[&'static str]) -> Result> { #[test] fn scheduler_basic() -> Result<()> { - insta::assert_yaml_snapshot!(generate_results(&["--runs=2", "sleep 0.123", "sleep 0.456"])?, @r#" + insta::assert_yaml_snapshot!(generate_results(&["--runs=2", "sleep 0.123", "sleep 0.456"])?, @" - command: sleep 0.123 measurements: - time_wall_clock: @@ -97,6 +97,9 @@ fn scheduler_basic() -> Result<()> { value: 0 unit: byte exit_code: 0 + time_cpu: + value: 0 + unit: second - time_wall_clock: value: 0.123 unit: second @@ -110,6 +113,9 @@ fn scheduler_basic() -> Result<()> { value: 0 unit: byte exit_code: 0 + time_cpu: + value: 0 + unit: second - command: sleep 0.456 measurements: - time_wall_clock: @@ -125,6 +131,9 @@ fn scheduler_basic() -> Result<()> { value: 0 unit: byte exit_code: 0 + time_cpu: + value: 0 + unit: second - time_wall_clock: value: 0.456 unit: second @@ -138,7 +147,10 @@ fn scheduler_basic() -> Result<()> { value: 0 unit: byte exit_code: 0 - "#); + time_cpu: + value: 0 + unit: second + "); Ok(()) } diff --git a/src/cli.rs b/src/cli.rs index cab618c..fb8ca3e 100644 --- a/src/cli.rs +++ b/src/cli.rs @@ -57,11 +57,11 @@ fn build_command() -> Command { .default_value("default") .help("Metrics to display, in order. The first is used for non-JSON exports. \ Compare all commands with the first command. Presets: default (wall-clock \ - time and peak RSS; wall-clock time only on Windows), speed (wall-clock, user, \ - and system time), all (all available metrics, skipping unavailable \ + time and peak RSS; wall-clock time only on Windows), time (wall-clock, total CPU, \ + user, and system time), all (all available metrics, skipping unavailable \ measurements). All presets use automatic \ units with wall-clock time first; use a preset on its own. Metrics: time_wall_clock, \ - time_user, time_system, memory_peak_resident, cpu_cycles, instructions, \ + time_cpu (user + system), time_user, time_system, memory_peak_resident, cpu_cycles, instructions, \ cache_references, cache_misses, branch_misses. Optional units: ns/us/ms/s/min/h \ for time, B/kB/MB/GB/TB/KiB/MiB/GiB/TiB for memory, count/k/M/B for counters \ (k = thousand, M = million, B = billion). Example: \ diff --git a/src/metric.rs b/src/metric.rs index da3547d..98fd76c 100644 --- a/src/metric.rs +++ b/src/metric.rs @@ -9,6 +9,7 @@ use crate::quantity::{byte, ratio, second, Ratio}; #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum Metric { TimeWallClock, + TimeCpu, TimeUser, TimeSystem, MemoryPeakResident, @@ -20,8 +21,9 @@ pub enum Metric { } impl Metric { - pub const ALL: [Self; 9] = [ + pub const ALL: [Self; 10] = [ Self::TimeWallClock, + Self::TimeCpu, Self::TimeUser, Self::TimeSystem, Self::MemoryPeakResident, @@ -35,6 +37,7 @@ impl Metric { pub fn name(self) -> &'static str { match self { Self::TimeWallClock => "time_wall_clock", + Self::TimeCpu => "time_cpu", Self::TimeUser => "time_user", Self::TimeSystem => "time_system", Self::MemoryPeakResident => "memory_peak_resident", @@ -49,6 +52,7 @@ impl Metric { pub fn label(self) -> &'static str { match self { Self::TimeWallClock => "Wall Time", + Self::TimeCpu => "CPU time", Self::TimeUser => "User time", Self::TimeSystem => "System time", Self::MemoryPeakResident => "Memory", @@ -65,6 +69,7 @@ impl Metric { let counters = &measurement.hardware_counters; match self { Self::TimeWallClock => Some(measurement.time_wall_clock.get::()), + Self::TimeCpu => Some(measurement.time_cpu().get::()), Self::TimeUser => Some(measurement.time_user.get::()), Self::TimeSystem => Some(measurement.time_system.get::()), Self::MemoryPeakResident => measurement.memory_peak_resident.map(|v| v.get::()), @@ -79,7 +84,7 @@ impl Metric { pub fn is_time(self) -> bool { matches!( self, - Self::TimeWallClock | Self::TimeUser | Self::TimeSystem + Self::TimeWallClock | Self::TimeCpu | Self::TimeUser | Self::TimeSystem ) } @@ -349,7 +354,33 @@ impl Stats { #[cfg(test)] mod tests { use super::*; - use crate::quantity::Information; + use crate::quantity::{Information, Time}; + + #[test] + fn cpu_time_statistics_use_per_run_totals() { + let measurements = Measurements::new( + [(1.0, 3.0), (3.0, 1.0)] + .iter() + .map(|&(user, system)| Measurement { + time_user: Time::new::(user), + time_system: Time::new::(system), + ..Measurement::default() + }) + .collect(), + ); + let selection = MetricSelection::parse_list("time_cpu:ms").unwrap()[0]; + let stats = selection.metric.summarize(&measurements).unwrap(); + assert_eq!(selection.csv_unit().format(stats.mean), "4000.0 ms"); + // User and system time vary, but their per-run total is constant. + assert_eq!(stats.stddev, Some(0.0)); + assert_eq!(stats.min, 4.0); + assert_eq!(stats.max, 4.0); + let json = serde_json::to_value(&measurements).unwrap(); + for sample in json["measurements"].as_array().unwrap() { + assert_eq!(sample["time_cpu"]["value"], 4.0); + assert_eq!(sample["time_cpu"]["unit"], "second"); + } + } #[test] fn selection_units_and_validation() { diff --git a/src/options.rs b/src/options.rs index e24ed3b..d2d4130 100644 --- a/src/options.rs +++ b/src/options.rs @@ -397,10 +397,15 @@ impl Options { match matches.get_one::("metrics").unwrap().as_str() { "default" => {} - "speed" => { - options.metrics = [Metric::TimeWallClock, Metric::TimeUser, Metric::TimeSystem] - .map(|metric| MetricSelection { metric, unit: None }) - .to_vec(); + "time" => { + options.metrics = [ + Metric::TimeWallClock, + Metric::TimeCpu, + Metric::TimeUser, + Metric::TimeSystem, + ] + .map(|metric| MetricSelection { metric, unit: None }) + .to_vec(); } "all" => { let shell = matches!( diff --git a/tests/integration_tests.rs b/tests/integration_tests.rs index 26cf9fc..47cef60 100644 --- a/tests/integration_tests.rs +++ b/tests/integration_tests.rs @@ -55,7 +55,7 @@ fn runs_successfully() { #[test] fn metric_presets_select_available_metrics() { - for preset in ["speed", "all"] { + for preset in ["time", "all"] { let output = hyperfine_debug() .arg(format!("--metrics={preset}")) .args(["--style=basic", "--runs=2", "sleep 1", "sleep 2"]) @@ -64,6 +64,7 @@ fn metric_presets_select_available_metrics() { assert!(output.status.success(), "{:?}", output); let stdout = String::from_utf8(output.stdout).unwrap(); assert_eq!(stdout.matches("Wall Time").count(), 2); + assert_eq!(stdout.matches("CPU time").count(), 2); assert_eq!(stdout.matches("User time").count(), 2); assert_eq!(stdout.matches("System time").count(), 2); assert_eq!(stdout.contains("Memory"), preset == "all" && !cfg!(windows)); diff --git a/tests/snapshots/integration_tests__json_export_basic.snap b/tests/snapshots/integration_tests__json_export_basic.snap index 63b0810..a3285ce 100644 --- a/tests/snapshots/integration_tests__json_export_basic.snap +++ b/tests/snapshots/integration_tests__json_export_basic.snap @@ -50,7 +50,11 @@ exit_code: 0 "value": 0.0, "unit": "byte" }, - "exit_code": 0 + "exit_code": 0, + "time_cpu": { + "value": 0.0, + "unit": "second" + } }, { "time_wall_clock": { @@ -69,7 +73,11 @@ exit_code: 0 "value": 0.0, "unit": "byte" }, - "exit_code": 0 + "exit_code": 0, + "time_cpu": { + "value": 0.0, + "unit": "second" + } } ], "summary": { @@ -82,6 +90,15 @@ exit_code: 0 "min": 0.0, "max": 0.0 }, + "time_cpu": { + "unit": "second", + "count": 2, + "mean": 0.0, + "stddev": 0.0, + "median": 0.0, + "min": 0.0, + "max": 0.0 + }, "time_system": { "unit": "second", "count": 2, @@ -131,7 +148,11 @@ exit_code: 0 "value": 0.0, "unit": "byte" }, - "exit_code": 0 + "exit_code": 0, + "time_cpu": { + "value": 0.0, + "unit": "second" + } }, { "time_wall_clock": { @@ -150,7 +171,11 @@ exit_code: 0 "value": 0.0, "unit": "byte" }, - "exit_code": 0 + "exit_code": 0, + "time_cpu": { + "value": 0.0, + "unit": "second" + } } ], "summary": { @@ -163,6 +188,15 @@ exit_code: 0 "min": 0.0, "max": 0.0 }, + "time_cpu": { + "unit": "second", + "count": 2, + "mean": 0.0, + "stddev": 0.0, + "median": 0.0, + "min": 0.0, + "max": 0.0 + }, "time_system": { "unit": "second", "count": 2, diff --git a/tests/snapshots/integration_tests__json_export_parameterized_with_reference.snap b/tests/snapshots/integration_tests__json_export_parameterized_with_reference.snap index 798d295..c9d05af 100644 --- a/tests/snapshots/integration_tests__json_export_parameterized_with_reference.snap +++ b/tests/snapshots/integration_tests__json_export_parameterized_with_reference.snap @@ -50,7 +50,11 @@ exit_code: 0 "value": 0.0, "unit": "byte" }, - "exit_code": 0 + "exit_code": 0, + "time_cpu": { + "value": 0.0, + "unit": "second" + } }, { "time_wall_clock": { @@ -69,7 +73,11 @@ exit_code: 0 "value": 0.0, "unit": "byte" }, - "exit_code": 0 + "exit_code": 0, + "time_cpu": { + "value": 0.0, + "unit": "second" + } } ], "parameters": { @@ -87,6 +95,15 @@ exit_code: 0 "min": 0.0, "max": 0.0 }, + "time_cpu": { + "unit": "second", + "count": 2, + "mean": 0.0, + "stddev": 0.0, + "median": 0.0, + "min": 0.0, + "max": 0.0 + }, "time_system": { "unit": "second", "count": 2, @@ -137,7 +154,11 @@ exit_code: 0 "value": 0.0, "unit": "byte" }, - "exit_code": 0 + "exit_code": 0, + "time_cpu": { + "value": 0.0, + "unit": "second" + } }, { "time_wall_clock": { @@ -156,7 +177,11 @@ exit_code: 0 "value": 0.0, "unit": "byte" }, - "exit_code": 0 + "exit_code": 0, + "time_cpu": { + "value": 0.0, + "unit": "second" + } } ], "parameters": { @@ -174,6 +199,15 @@ exit_code: 0 "min": 0.0, "max": 0.0 }, + "time_cpu": { + "unit": "second", + "count": 2, + "mean": 0.0, + "stddev": 0.0, + "median": 0.0, + "min": 0.0, + "max": 0.0 + }, "time_system": { "unit": "second", "count": 2, diff --git a/tests/snapshots/integration_tests__json_export_single_run.snap b/tests/snapshots/integration_tests__json_export_single_run.snap index e94300e..953dcd6 100644 --- a/tests/snapshots/integration_tests__json_export_single_run.snap +++ b/tests/snapshots/integration_tests__json_export_single_run.snap @@ -45,7 +45,11 @@ exit_code: 0 "value": 0.0, "unit": "byte" }, - "exit_code": 0 + "exit_code": 0, + "time_cpu": { + "value": 0.0, + "unit": "second" + } } ], "summary": { @@ -58,6 +62,15 @@ exit_code: 0 "min": 0.0, "max": 0.0 }, + "time_cpu": { + "unit": "second", + "count": 1, + "mean": 0.0, + "stddev": null, + "median": 0.0, + "min": 0.0, + "max": 0.0 + }, "time_system": { "unit": "second", "count": 1,