Updates

CHANGELOG.md

@@ -2,7 +2,7 @@
 
 ## Breaking changes
 
-- Replace `--time-unit`/`-u` with `--metrics METRIC[:UNIT],…`. Select multiple metrics for terminal output; the first metric is primary for all non-JSON exports. Presets include `default`, `speed` (wall-clock, user, and system time), and `all` (all available metrics). Units are chosen automatically for terminal and markup output when omitted. Counters support decimal `k`, `M`, and `B` suffixes.
+- Replace `--time-unit`/`-u` with `--metrics METRIC[:UNIT],…`. Select multiple metrics for terminal output; the first metric is primary for all non-JSON exports. Presets include `default`, `time` (wall-clock, total CPU, user, and system time), and `all` (all available metrics). The `time_cpu` metric sums user and system time for each run. Units are chosen automatically for terminal and markup output when omitted. Counters support decimal `k`, `M`, and `B` suffixes.
 - Remove `--sort`, `--reference`, and `--reference-name`. Commands and exports always remain in input order, and the first expanded benchmark is the reference for every metric.
 - Replace time-only terminal output and the final speed ranking with a per-command metric table. Show wall-clock time and peak resident memory by default (wall-clock time only on Windows), with percentage changes from the first command for subsequent benchmarks. Color all metric rows and make the entire primary row bold.
 - Export only the primary metric to CSV, Markdown, AsciiDoc, and org-mode. CSV uses an explicitly selected unit or the metric's base unit when omitted. JSON continues to contain all collected metrics in base units and identifies the primary metric.

README.md

@@ -58,9 +58,9 @@ hyperfine \
     './baseline' './candidate'
 ```
 
-Use `--metrics=default` for the platform defaults, `--metrics=speed` for
-`time_wall_clock,time_user,time_system`, or `--metrics=all` for all available metrics in the
-order listed below. The `speed` preset works on all platforms, including with a shell.
+Use `--metrics=default` for the platform defaults, `--metrics=time` for
+`time_wall_clock,time_cpu,time_user,time_system`, or `--metrics=all` for all available metrics in the
+order listed below. The `time` preset works on all platforms, including with a shell.
 All presets keep wall-clock time primary, choose display units automatically, and must be
 used alone, without other metrics or unit suffixes.
 The `all` preset skips metrics unsupported by the platform or shell, or missing from any run
@@ -81,7 +81,7 @@ separate reference selection.
 
 | Metrics | Explicit units |
 |:---|:---|
-| `time_wall_clock`, `time_user`, `time_system` | `ns`, `us`, `ms`, `s`, `min`, `h` |
+| `time_wall_clock`, `time_cpu`, `time_user`, `time_system` | `ns`, `us`, `ms`, `s`, `min`, `h` |
 | `memory_peak_resident` | `B`, `kB`, `MB`, `GB`, `TB`, `KiB`, `MiB`, `GiB`, `TiB` |
 | `cpu_cycles`, `instructions`, `cache_references`, `cache_misses`, `branch_misses` | `count`, `k`, `M`, `B` |
 
@@ -234,17 +234,23 @@ The metrics are:
 
 - **`time_wall_clock`**: Time from start to finish, including time spent waiting, in seconds.
 
-- **`time_user`**: CPU time spent running the program's code, summed across threads, in seconds.
+- **`time_user`**: CPU time spent executing application and library code in user mode,
+  summed across threads, in seconds.
 
   - Linux/macOS: Includes child-process time when parents wait for their children to finish.
   - Windows: Includes the command and its child processes.
 
-- **`time_system`**: CPU time spent running operating-system code for the program, for example
+- **`time_system`**: CPU time spent executing kernel code on the program's behalf, for example
   to read files, summed across threads, in seconds.
 
   - Linux/macOS: Includes child-process time when parents wait for their children to finish.
   - Windows: Includes the command and its child processes.
 
+- **`time_cpu`**: Total CPU time, calculated as `time_user + time_system` for each run, in seconds.
+  It excludes time spent sleeping or waiting without executing. CPU time is summed across
+  threads, so it can exceed wall-clock time: four threads running on four cores for one second
+  can consume roughly four CPU-seconds.
+
 - **`memory_peak_resident`**: Peak memory held in physical RAM, in bytes.
 
   - Linux/macOS: Peak resident set size (RSS). This is the largest per-process peak among the

doc/hyperfine.1

@@ -35,7 +35,7 @@ hyperfine \- command\-line benchmarking tool
 .RB [ \-\-style
 .IR TYPE ]
 .RB [ \-\-metrics
-.IR METRIC[:UNIT],...|default|speed|all ]
+.IR METRIC[:UNIT],...|default|time|all ]
 .RB [ \-\-export\-asciidoc
 .IR FILE ]
 .RB [ \-\-export\-csv
@@ -238,7 +238,7 @@ interactive output without any colors. Set this to 'color' to keep the colors
 without any interactive output. Set this to 'none' to disable all the output
 of the tool.
 .HP
-\fB\-\-metrics\fR \fIMETRIC[:UNIT],...|default|speed|all\fP
+\fB\-\-metrics\fR \fIMETRIC[:UNIT],...|default|time|all\fP
 .IP
 Select the metrics to display, in order, with an optional unit for each.
 Default: \fBtime_wall_clock,memory_peak_resident\fR, or \fBtime_wall_clock\fR
@@ -249,9 +249,9 @@ The first benchmark has no change column; subsequent benchmarks show percentage
 changes from the first benchmark. Decreases are green and increases are red
 when colors are enabled.
 .IP
-Use \fBdefault\fR for the platform defaults, \fBspeed\fR for
-\fBtime_wall_clock,time_user,time_system\fR, or \fBall\fR for all available
-metrics in the order listed below. The \fBspeed\fR preset works on all
+Use \fBdefault\fR for the platform defaults, \fBtime\fR for
+\fBtime_wall_clock,time_cpu,time_user,time_system\fR, or \fBall\fR for all available
+metrics in the order listed below. The \fBtime\fR preset works on all
 platforms, including with a shell.
 All presets keep wall\-clock time primary, select display units automatically,
 and must be used alone, without other metrics or unit suffixes.
@@ -259,8 +259,10 @@ The \fBall\fR preset skips metrics unsupported by the platform or shell,
 or missing from any run of a benchmark. If a metric is unavailable for the
 reference, its change displays as N/A.
 .IP
-Time metrics \fBtime_wall_clock\fR, \fBtime_user\fR, and \fBtime_system\fR
+Time metrics \fBtime_wall_clock\fR, \fBtime_cpu\fR, \fBtime_user\fR, and \fBtime_system\fR
 accept units \fBns\fR, \fBus\fR, \fBms\fR, \fBs\fR, \fBmin\fR, and \fBh\fR.
+Total CPU time, \fBtime_cpu\fR, is the sum of user and system time for each run.
+It is summed across threads and can exceed wall\-clock time when work runs in parallel.
 Peak memory, \fBmemory_peak_resident\fR, accepts \fBB\fR, \fBkB\fR,
 \fBMB\fR, \fBGB\fR, \fBTB\fR, \fBKiB\fR, \fBMiB\fR, \fBGiB\fR,
 and \fBTiB\fR.

scripts/README.md

@@ -7,14 +7,32 @@ hyperfine 'sleep 0.020' 'sleep 0.021' 'sleep 0.022' --export-json sleep.json
 uv run plot_whisker.py sleep.json
 ```
 
-All plotting scripts accept `--metric` to select a metric. Hardware counters
-(`cpu_cycles`, `instructions`, `cache_references`, `cache_misses`, and `branch_misses`)
-are plotted as unscaled counts when available in the JSON export. For example:
+All plotting and analysis scripts use the JSON export's `primary_metric` by default.
+Pass `--metric` to select a different metric. Supported metrics are `time_wall_clock`,
+`time_cpu`, `time_user`, `time_system`, `memory_peak_resident`, `cpu_cycles`,
+`instructions`, `cache_references`, `cache_misses`, and `branch_misses`, provided the
+selected metric is available in every benchmark being analyzed. For example:
 
 ```bash
 uv run plot_histogram.py benchmark.json --metric instructions
 ```
 
+Time is displayed in seconds, memory in MiB, and counters as unscaled counts.
+Multi-file plots require matching primary metrics unless `--metric` explicitly selects
+one metric across all files. Exports without `primary_metric` fall back to wall-clock
+time; the per-run `measurements` JSON schema is still required.
+
+For descriptive statistics or a Welch's t-test comparing exactly two benchmark results:
+
+```bash
+uv run advanced_statistics.py benchmark.json --metric time_cpu --time-unit millisecond
+uv run welch_ttest.py comparison.json --metric memory_peak_resident
+```
+
+`advanced_statistics.py` accepts `--time-unit second` or `--time-unit millisecond`
+for time metrics only. Both analysis scripts also accept the default primary metric
+without `--metric`.
+
 ### Prerequisites
 
 Install [`uv`](https://docs.astral.sh/uv/getting-started/installation/) and run the

scripts/advanced_statistics.py

@@ -7,74 +7,56 @@
 # ///
 
 import argparse
-import json
-from enum import Enum
 
 import numpy as np
 
-
-class Unit(Enum):
-    SECOND = 1
-    MILLISECOND = 2
-
-    def factor(self):
-        match self:
-            case Unit.SECOND:
-                return 1
-            case Unit.MILLISECOND:
-                return 1e3
-
-    def __str__(self):
-        match self:
-            case Unit.SECOND:
-                return "s"
-            case Unit.MILLISECOND:
-                return "ms"
-
+from plot_utils import METRICS, add_metric_argument, load_results
 
 parser = argparse.ArgumentParser()
 parser.add_argument("file", help="JSON file with benchmark results")
+add_metric_argument(parser)
 parser.add_argument(
     "--time-unit",
-    help="The unit of time.",
-    default="second",
-    action="store",
+    help="Display unit for time metrics (default: second).",
     choices=["second", "millisecond"],
-    dest="unit",
 )
 args = parser.parse_args()
 
-unit = Unit.MILLISECOND if args.unit == "millisecond" else Unit.SECOND
-unit_str = str(unit)
+metric, [results] = load_results(parser, [args.file], args.metric)
+description = METRICS[metric]
+unit, scale = description.unit, description.scale
+if args.time_unit is not None:
+    if not metric.startswith("time_"):
+        parser.error("--time-unit can only be used with a time metric")
+    if args.time_unit == "millisecond":
+        unit, scale = "ms", 0.001
 
-with open(args.file) as f:
-    results = json.load(f)["results"]
 
+def format_value(value):
+    suffix = f" {unit}" if unit else ""
+    return f"{value:.3f}{suffix}"
+
+
+print(f"Metric: {description.label}\n")
 for result in results:
-    summary = result["summary"]["time_wall_clock"]
-    ts = [m["time_wall_clock"]["value"] * unit.factor() for m in result["measurements"]]
+    summary = result["summary"][metric]
+    values = [m[metric]["value"] / scale for m in result["measurements"]]
 
-    p05 = np.percentile(ts, 5)
-    p25 = np.percentile(ts, 25)
-    p75 = np.percentile(ts, 75)
-    p95 = np.percentile(ts, 95)
+    p05, p25, p75, p95 = np.percentile(values, [5, 25, 75, 95])
 
     iqr = p75 - p25
 
     print(f"Command '{result.get('name', result['command'])}'")
     print(f"  runs:   {summary['count']:8d}")
-    print(f"  mean:   {summary['mean'] * unit.factor():8.3f} {unit_str}")
-    if summary["stddev"] is None:
-        print(f"  stddev: {'N/A':>8}")
-    else:
-        print(f"  stddev: {summary['stddev'] * unit.factor():8.3f} {unit_str}")
-    print(f"  median: {summary['median'] * unit.factor():8.3f} {unit_str}")
-    print(f"  min:    {summary['min'] * unit.factor():8.3f} {unit_str}")
-    print(f"  max:    {summary['max'] * unit.factor():8.3f} {unit_str}")
+    for statistic in ["mean", "stddev", "median", "min", "max"]:
+        value = summary[statistic]
+        formatted = "N/A" if value is None else format_value(value / scale)
+        print(f"  {statistic + ':':<8}{formatted:>8}")
     print()
     print("  percentiles:")
-    print(f"     P_05 .. P_95:    {p05:.3f} {unit_str} .. {p95:.3f} {unit_str}")
+    print(f"     P_05 .. P_95:    {format_value(p05)} .. {format_value(p95)}")
     print(
-        f"     P_25 .. P_75:    {p25:.3f} {unit_str} .. {p75:.3f} {unit_str}  (IQR = {iqr:.3f} {unit_str})"
+        f"     P_25 .. P_75:    {format_value(p25)} .. {format_value(p75)}"
+        f"  (IQR = {format_value(iqr)})"
     )
     print()

scripts/plot_benchmark_comparison.py

@@ -14,13 +14,12 @@ Note all the input files must contain results for all commands.
 """
 
 import argparse
-import json
 import pathlib
 
 import matplotlib.pyplot as plt
 import numpy as np
 
-from plot_utils import METRICS, add_metric_argument, validate_metric
+from plot_utils import METRICS, add_metric_argument, load_results
 
 parser = argparse.ArgumentParser(description=__doc__)
 add_metric_argument(parser)
@@ -34,7 +33,8 @@ parser.add_argument(
 parser.add_argument("-o", "--output", help="Save image to the given filename")
 
 args = parser.parse_args()
-metric_label, metric_scale = METRICS[args.metric]
+metric, datasets = load_results(parser, args.files, args.metric)
+metric_info = METRICS[metric]
 
 commands = None
 data = []
@@ -45,10 +45,7 @@ if args.benchmark_names:
         "Number of benchmark names must match the number of input files."
     )
 
-for i, filename in enumerate(args.files):
-    with open(filename) as f:
-        results = json.load(f)["results"]
-    validate_metric(parser, results, args.metric)
+for i, (filename, results) in enumerate(zip(args.files, datasets)):
     benchmark_commands = [b.get("name", b["command"]) for b in results]
     if commands is None:
         commands = benchmark_commands
@@ -56,7 +53,7 @@ for i, filename in enumerate(args.files):
         assert commands == benchmark_commands, (
             f"Unexpected commands in {filename}: {benchmark_commands}, expected: {commands}"
         )
-    data.append([b["summary"][args.metric]["mean"] / metric_scale for b in results])
+    data.append([b["summary"][metric]["mean"] / metric_info.scale for b in results])
     if args.benchmark_names:
         inputs.append(args.benchmark_names[i])
     else:
@@ -79,7 +76,7 @@ ax.grid(visible=True, axis="y")
 if args.title:
     plt.title(args.title)
 plt.xlabel("Benchmark")
-plt.ylabel(metric_label)
+plt.ylabel(metric_info.axis_label)
 plt.legend(title="Command")
 
 if args.output:

scripts/plot_histogram.py

@@ -11,12 +11,11 @@
 """This program shows `hyperfine` benchmark results as a histogram."""
 
 import argparse
-import json
 
 import matplotlib.pyplot as plt
 import numpy as np
 
-from plot_utils import METRICS, add_metric_argument, validate_metric
+from plot_utils import METRICS, add_metric_argument, load_results
 
 parser = argparse.ArgumentParser(description=__doc__)
 add_metric_argument(parser)
@@ -70,18 +69,15 @@ parser.add_argument(
 )
 
 args = parser.parse_args()
-metric_label, metric_scale = METRICS[args.metric]
-
-with open(args.file) as f:
-    results = json.load(f)["results"]
-validate_metric(parser, results, args.metric)
+metric, [results] = load_results(parser, [args.file], args.metric)
+metric_info = METRICS[metric]
 
 if args.labels:
     labels = args.labels.split(",")
 else:
     labels = [b.get("name", b["command"]) for b in results]
 all_values = [
-    [m[args.metric]["value"] / metric_scale for m in b["measurements"]] for b in results
+    [m[metric]["value"] / metric_info.scale for m in b["measurements"]] for b in results
 ]
 
 value_min = (
@@ -113,7 +109,7 @@ plt.legend(
     prop={"size": 10, "family": ["Source Code Pro", "Fira Mono", "Courier New"]},
 )
 
-plt.xlabel(metric_label)
+plt.xlabel(metric_info.axis_label)
 if args.title:
     plt.title(args.title)
 

scripts/plot_parametrized.py

@@ -11,12 +11,11 @@
 errorbar plot."""
 
 import argparse
-import json
 import sys
 
 import matplotlib.pyplot as plt
 
-from plot_utils import METRICS, add_metric_argument, validate_metric
+from plot_utils import METRICS, add_metric_argument, load_results
 
 parser = argparse.ArgumentParser(description=__doc__)
 add_metric_argument(parser)
@@ -43,7 +42,8 @@ parser.add_argument(
 parser.add_argument("-o", "--output", help="Save image to the given filename.")
 
 args = parser.parse_args()
-metric_label, metric_scale = METRICS[args.metric]
+metric, datasets = load_results(parser, args.file, args.metric)
+metric_info = METRICS[metric]
 if args.parameter_name is not None:
     sys.stderr.write(
         "warning: --parameter-name is deprecated; names are inferred from "
@@ -84,11 +84,7 @@ def unique_parameter(benchmark):
 
 parameter_name = None
 
-for filename in args.file:
-    with open(filename) as f:
-        results = json.load(f)["results"]
-    validate_metric(parser, results, args.metric)
-
+for results in datasets:
     (this_parameter_name, parameter_values) = extract_parameters(results)
     if parameter_name is not None and this_parameter_name != parameter_name:
         die(
@@ -96,17 +92,17 @@ for filename in args.file:
         )
     parameter_name = this_parameter_name
 
-    summaries = [b["summary"][args.metric] for b in results]
-    means = [s["mean"] / metric_scale for s in summaries]
+    summaries = [b["summary"][metric] for b in results]
+    means = [s["mean"] / metric_info.scale for s in summaries]
     stddevs = [
-        s["stddev"] / metric_scale if s["stddev"] is not None else float("nan")
+        s["stddev"] / metric_info.scale if s["stddev"] is not None else float("nan")
         for s in summaries
     ]
 
     plt.errorbar(x=parameter_values, y=means, yerr=stddevs, capsize=2)
 
 plt.xlabel(parameter_name)
-plt.ylabel(metric_label)
+plt.ylabel(metric_info.axis_label)
 
 if args.log_y:
     plt.yscale("log")

scripts/plot_progression.py

@@ -14,12 +14,11 @@ thermal throttling and similar effects.
 """
 
 import argparse
-import json
 
 import matplotlib.pyplot as plt
 import numpy as np
 
-from plot_utils import METRICS, add_metric_argument, validate_metric
+from plot_utils import METRICS, add_metric_argument, load_results
 
 
 def moving_average(values, num_runs):
@@ -50,15 +49,12 @@ parser.add_argument(
 
 
 args = parser.parse_args()
-metric_label, metric_scale = METRICS[args.metric]
-
-with open(args.file) as f:
-    results = json.load(f)["results"]
-validate_metric(parser, results, args.metric)
+metric, [results] = load_results(parser, [args.file], args.metric)
+metric_info = METRICS[metric]
 
 for result in results:
     label = result.get("name", result["command"])
-    values = [m[args.metric]["value"] / metric_scale for m in result["measurements"]]
+    values = [m[metric]["value"] / metric_info.scale for m in result["measurements"]]
     num = len(values)
     nums = range(num)
 
@@ -86,7 +82,7 @@ for result in results:
         legend.append("moving average")
 plt.legend(legend)
 
-plt.ylabel(metric_label)
+plt.ylabel(metric_info.axis_label)
 
 if args.output:
     plt.savefig(args.output)

scripts/plot_utils.py

@@ -1,15 +1,30 @@
-"""Shared metric options for the plotting scripts."""
+"""Shared metric selection and units for analysis and plotting scripts."""
+
+import json
+from typing import NamedTuple
+
+
+class Metric(NamedTuple):
+    label: str
+    unit: str = ""
+    scale: float = 1
+
+    @property
+    def axis_label(self):
+        return f"{self.label} [{self.unit}]" if self.unit else self.label
+
 
 METRICS = {
-    "time_wall_clock": ("Time [s]", 1),
-    "time_user": ("User CPU time [s]", 1),
-    "time_system": ("System CPU time [s]", 1),
-    "memory_peak_resident": ("Peak resident memory [MiB]", 1024**2),
-    "cpu_cycles": ("CPU cycles", 1),
-    "instructions": ("Completed CPU instructions", 1),
-    "cache_references": ("Cache references", 1),
-    "cache_misses": ("Cache misses", 1),
-    "branch_misses": ("Mispredicted branches", 1),
+    "time_wall_clock": Metric("Wall-clock time", "s"),
+    "time_cpu": Metric("Total CPU time", "s"),
+    "time_user": Metric("User CPU time", "s"),
+    "time_system": Metric("System CPU time", "s"),
+    "memory_peak_resident": Metric("Peak resident memory", "MiB", 1024**2),
+    "cpu_cycles": Metric("CPU cycles"),
+    "instructions": Metric("Completed CPU instructions"),
+    "cache_references": Metric("Cache references"),
+    "cache_misses": Metric("Cache misses"),
+    "branch_misses": Metric("Mispredicted branches"),
 }
 
 
@@ -17,8 +32,8 @@ def add_metric_argument(parser):
     parser.add_argument(
         "--metric",
         choices=METRICS,
-        default="time_wall_clock",
-        help="Metric to plot (default: time_wall_clock). "
+        help="Metric to analyze (default: primary_metric from the JSON export, "
+        "or time_wall_clock if unspecified). "
         "Times are in seconds; memory is in MiB; hardware counters are unscaled counts.",
     )
 
@@ -30,3 +45,30 @@ def validate_metric(parser, results, metric):
                 f"metric {metric!r} is unavailable for "
                 f"{result.get('name', result['command'])!r}"
             )
+
+
+def load_results(parser, filenames, metric=None):
+    exports = []
+    for filename in filenames:
+        with open(filename, encoding="utf-8") as f:
+            export = json.load(f)
+        if not export["results"]:
+            parser.error(f"no benchmark results in {filename}")
+        exports.append(export)
+
+    if metric is None:
+        primary_metrics = {
+            export.get("primary_metric", "time_wall_clock") for export in exports
+        }
+        if len(primary_metrics) != 1:
+            parser.error(
+                "input files use different primary metrics; select one with --metric"
+            )
+        metric = primary_metrics.pop()
+    if metric not in METRICS:
+        parser.error(f"unknown metric {metric!r}; select one with --metric")
+
+    results = [export["results"] for export in exports]
+    for benchmarks in results:
+        validate_metric(parser, benchmarks, metric)
+    return metric, results

scripts/plot_whisker.py

@@ -16,11 +16,10 @@ Quoting from the matplotlib documentation:
 """
 
 import argparse
-import json
 
 import matplotlib.pyplot as plt
 
-from plot_utils import METRICS, add_metric_argument, validate_metric
+from plot_utils import METRICS, add_metric_argument, load_results
 
 parser = argparse.ArgumentParser(description=__doc__)
 add_metric_argument(parser)
@@ -33,22 +32,19 @@ parser.add_argument(
 parser.add_argument("-o", "--output", help="Save image to the given filename.")
 
 args = parser.parse_args()
-metric_label, metric_scale = METRICS[args.metric]
-
-with open(args.file, encoding="utf-8") as f:
-    results = json.load(f)["results"]
-validate_metric(parser, results, args.metric)
+metric, [results] = load_results(parser, [args.file], args.metric)
+metric_info = METRICS[metric]
 
 if args.labels:
     labels = args.labels.split(",")
 else:
     labels = [b.get("name", b["command"]) for b in results]
 values = [
-    [m[args.metric]["value"] / metric_scale for m in b["measurements"]] for b in results
+    [m[metric]["value"] / metric_info.scale for m in b["measurements"]] for b in results
 ]
 
 if args.sort_by == "median":
-    medians = [b["summary"][args.metric]["median"] for b in results]
+    medians = [b["summary"][metric]["median"] for b in results]
     indices = sorted(range(len(labels)), key=lambda k: medians[k])
     labels = [labels[i] for i in indices]
     values = [values[i] for i in indices]
@@ -64,7 +60,7 @@ for patch, color in zip(boxplot["boxes"], colors):
 if args.title:
     plt.title(args.title)
 plt.legend(handles=boxplot["boxes"], labels=labels, loc="best", fontsize="medium")
-plt.ylabel(metric_label)
+plt.ylabel(metric_info.axis_label)
 plt.ylim(0, None)
 plt.xticks(list(range(1, len(labels) + 1)), labels, rotation=45)
 if args.output:

scripts/welch_ttest.py

@@ -7,32 +7,37 @@
 # ///
 
 """This script performs Welch's t-test on a JSON export file with two
-benchmark results to test for a difference in their population mean runtimes."""
+benchmark results to test for a difference in their population means."""
 
 import argparse
-import json
-import sys
+import math
 
 from scipy import stats
 
+from plot_utils import METRICS, add_metric_argument, load_results
+
 parser = argparse.ArgumentParser(description=__doc__)
 parser.add_argument("file", help="JSON file with two benchmark results")
+add_metric_argument(parser)
 args = parser.parse_args()
 
-with open(args.file) as f:
-    results = json.load(f)["results"]
+metric, [results] = load_results(parser, [args.file], args.metric)
 
 if len(results) != 2:
-    print("The input file has to contain exactly two benchmarks")
-    sys.exit(1)
+    parser.error("the input file must contain exactly two benchmarks")
 
 a, b = (x.get("name", x["command"]) for x in results[:2])
-X, Y = ([m["time_wall_clock"]["value"] for m in x["measurements"]] for x in results[:2])
+X, Y = ([m[metric]["value"] for m in x["measurements"]] for x in results[:2])
+if min(len(X), len(Y)) < 2:
+    parser.error("Welch's t-test requires at least two measurements per benchmark")
 
+print(f"Metric: {METRICS[metric].label}")
 print(f"Command 1: {a}")
 print(f"Command 2: {b}\n")
 
 t, p = stats.ttest_ind(X, Y, equal_var=False)
+if not math.isfinite(p):
+    parser.error("Welch's t-test is undefined for these measurements")
 th = 0.05
 dispose = p < th
 print(f"t = {t:.3}, p = {p:.3}")
@@ -40,9 +45,9 @@ print()
 
 if dispose:
     print(
-        f"A statistically significant difference in mean runtime was detected (p < {th})."
+        f"A statistically significant difference in mean value was detected (p < {th})."
     )
 else:
     print(
-        f"No statistically significant difference in mean runtime was detected (p >= {th})."
+        f"No statistically significant difference in mean value was detected (p >= {th})."
     )

src/benchmark/measurement.rs

@@ -100,8 +100,34 @@ pub struct Measurement {
     pub exit_status: ExitStatus,
 }
 
+impl Measurement {
+    /// Total CPU time for this run, including user and kernel mode.
+    pub fn time_cpu(&self) -> Time {
+        self.time_user + self.time_system
+    }
+}
+
+fn serialize_measurements<S>(measurements: &[Measurement], serializer: S) -> Result<S::Ok, S::Error>
+where
+    S: serde::Serializer,
+{
+    #[derive(Serialize)]
+    struct WithCpuTime<'a> {
+        #[serde(flatten)]
+        measurement: &'a Measurement,
+        #[serde(serialize_with = "serialize_time")]
+        time_cpu: Time,
+    }
+
+    serializer.collect_seq(measurements.iter().map(|measurement| WithCpuTime {
+        measurement,
+        time_cpu: measurement.time_cpu(),
+    }))
+}
+
 #[derive(Debug, Default, Clone, Serialize, PartialEq)]
 pub struct Measurements {
+    #[serde(serialize_with = "serialize_measurements")]
     pub measurements: Vec<Measurement>,
 }
 

src/benchmark/scheduler.rs

@@ -81,7 +81,7 @@ fn generate_results(args: &[&'static str]) -> Result<Vec<BenchmarkResult>> {
 
 #[test]
 fn scheduler_basic() -> Result<()> {
-    insta::assert_yaml_snapshot!(generate_results(&["--runs=2", "sleep 0.123", "sleep 0.456"])?, @r#"
+    insta::assert_yaml_snapshot!(generate_results(&["--runs=2", "sleep 0.123", "sleep 0.456"])?, @"
     - command: sleep 0.123
       measurements:
         - time_wall_clock:
@@ -97,6 +97,9 @@ fn scheduler_basic() -> Result<()> {
             value: 0
             unit: byte
           exit_code: 0
+          time_cpu:
+            value: 0
+            unit: second
         - time_wall_clock:
             value: 0.123
             unit: second
@@ -110,6 +113,9 @@ fn scheduler_basic() -> Result<()> {
             value: 0
             unit: byte
           exit_code: 0
+          time_cpu:
+            value: 0
+            unit: second
     - command: sleep 0.456
       measurements:
         - time_wall_clock:
@@ -125,6 +131,9 @@ fn scheduler_basic() -> Result<()> {
             value: 0
             unit: byte
           exit_code: 0
+          time_cpu:
+            value: 0
+            unit: second
         - time_wall_clock:
             value: 0.456
             unit: second
@@ -138,7 +147,10 @@ fn scheduler_basic() -> Result<()> {
             value: 0
             unit: byte
           exit_code: 0
-    "#);
+          time_cpu:
+            value: 0
+            unit: second
+    ");
 
     Ok(())
 }

src/cli.rs

@@ -57,11 +57,11 @@ fn build_command() -> Command {
                 .default_value("default")
                 .help("Metrics to display, in order. The first is used for non-JSON exports. \
                        Compare all commands with the first command. Presets: default (wall-clock \
-                       time and peak RSS; wall-clock time only on Windows), speed (wall-clock, user, \
-                       and system time), all (all available metrics, skipping unavailable \
+                       time and peak RSS; wall-clock time only on Windows), time (wall-clock, total CPU, \
+                       user, and system time), all (all available metrics, skipping unavailable \
                        measurements). All presets use automatic \
                        units with wall-clock time first; use a preset on its own. Metrics: time_wall_clock, \
-                       time_user, time_system, memory_peak_resident, cpu_cycles, instructions, \
+                       time_cpu (user + system), time_user, time_system, memory_peak_resident, cpu_cycles, instructions, \
                        cache_references, cache_misses, branch_misses. Optional units: ns/us/ms/s/min/h \
                        for time, B/kB/MB/GB/TB/KiB/MiB/GiB/TiB for memory, count/k/M/B for counters \
                        (k = thousand, M = million, B = billion). Example: \

src/metric.rs

@@ -9,6 +9,7 @@ use crate::quantity::{byte, ratio, second, Ratio};
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
 pub enum Metric {
     TimeWallClock,
+    TimeCpu,
     TimeUser,
     TimeSystem,
     MemoryPeakResident,
@@ -20,8 +21,9 @@ pub enum Metric {
 }
 
 impl Metric {
-    pub const ALL: [Self; 9] = [
+    pub const ALL: [Self; 10] = [
         Self::TimeWallClock,
+        Self::TimeCpu,
         Self::TimeUser,
         Self::TimeSystem,
         Self::MemoryPeakResident,
@@ -35,6 +37,7 @@ impl Metric {
     pub fn name(self) -> &'static str {
         match self {
             Self::TimeWallClock => "time_wall_clock",
+            Self::TimeCpu => "time_cpu",
             Self::TimeUser => "time_user",
             Self::TimeSystem => "time_system",
             Self::MemoryPeakResident => "memory_peak_resident",
@@ -49,6 +52,7 @@ impl Metric {
     pub fn label(self) -> &'static str {
         match self {
             Self::TimeWallClock => "Wall Time",
+            Self::TimeCpu => "CPU time",
             Self::TimeUser => "User time",
             Self::TimeSystem => "System time",
             Self::MemoryPeakResident => "Memory",
@@ -65,6 +69,7 @@ impl Metric {
         let counters = &measurement.hardware_counters;
         match self {
             Self::TimeWallClock => Some(measurement.time_wall_clock.get::<second>()),
+            Self::TimeCpu => Some(measurement.time_cpu().get::<second>()),
             Self::TimeUser => Some(measurement.time_user.get::<second>()),
             Self::TimeSystem => Some(measurement.time_system.get::<second>()),
             Self::MemoryPeakResident => measurement.memory_peak_resident.map(|v| v.get::<byte>()),
@@ -79,7 +84,7 @@ impl Metric {
     pub fn is_time(self) -> bool {
         matches!(
             self,
-            Self::TimeWallClock | Self::TimeUser | Self::TimeSystem
+            Self::TimeWallClock | Self::TimeCpu | Self::TimeUser | Self::TimeSystem
         )
     }
 
@@ -349,7 +354,33 @@ impl Stats {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use crate::quantity::Information;
+    use crate::quantity::{Information, Time};
+
+    #[test]
+    fn cpu_time_statistics_use_per_run_totals() {
+        let measurements = Measurements::new(
+            [(1.0, 3.0), (3.0, 1.0)]
+                .iter()
+                .map(|&(user, system)| Measurement {
+                    time_user: Time::new::<second>(user),
+                    time_system: Time::new::<second>(system),
+                    ..Measurement::default()
+                })
+                .collect(),
+        );
+        let selection = MetricSelection::parse_list("time_cpu:ms").unwrap()[0];
+        let stats = selection.metric.summarize(&measurements).unwrap();
+        assert_eq!(selection.csv_unit().format(stats.mean), "4000.0 ms");
+        // User and system time vary, but their per-run total is constant.
+        assert_eq!(stats.stddev, Some(0.0));
+        assert_eq!(stats.min, 4.0);
+        assert_eq!(stats.max, 4.0);
+        let json = serde_json::to_value(&measurements).unwrap();
+        for sample in json["measurements"].as_array().unwrap() {
+            assert_eq!(sample["time_cpu"]["value"], 4.0);
+            assert_eq!(sample["time_cpu"]["unit"], "second");
+        }
+    }
 
     #[test]
     fn selection_units_and_validation() {

src/options.rs

@@ -397,10 +397,15 @@ impl Options {
 
         match matches.get_one::<String>("metrics").unwrap().as_str() {
             "default" => {}
-            "speed" => {
-                options.metrics = [Metric::TimeWallClock, Metric::TimeUser, Metric::TimeSystem]
-                    .map(|metric| MetricSelection { metric, unit: None })
-                    .to_vec();
+            "time" => {
+                options.metrics = [
+                    Metric::TimeWallClock,
+                    Metric::TimeCpu,
+                    Metric::TimeUser,
+                    Metric::TimeSystem,
+                ]
+                .map(|metric| MetricSelection { metric, unit: None })
+                .to_vec();
             }
             "all" => {
                 let shell = matches!(

tests/integration_tests.rs

@@ -55,7 +55,7 @@ fn runs_successfully() {
 
 #[test]
 fn metric_presets_select_available_metrics() {
-    for preset in ["speed", "all"] {
+    for preset in ["time", "all"] {
         let output = hyperfine_debug()
             .arg(format!("--metrics={preset}"))
             .args(["--style=basic", "--runs=2", "sleep 1", "sleep 2"])
@@ -64,6 +64,7 @@ fn metric_presets_select_available_metrics() {
         assert!(output.status.success(), "{:?}", output);
         let stdout = String::from_utf8(output.stdout).unwrap();
         assert_eq!(stdout.matches("Wall Time").count(), 2);
+        assert_eq!(stdout.matches("CPU time").count(), 2);
         assert_eq!(stdout.matches("User time").count(), 2);
         assert_eq!(stdout.matches("System time").count(), 2);
         assert_eq!(stdout.contains("Memory"), preset == "all" && !cfg!(windows));

tests/snapshots/integration_tests__json_export_basic.snap

@@ -50,7 +50,11 @@ exit_code: 0
             "value": 0.0,
             "unit": "byte"
           },
-          "exit_code": 0
+          "exit_code": 0,
+          "time_cpu": {
+            "value": 0.0,
+            "unit": "second"
+          }
         },
         {
           "time_wall_clock": {
@@ -69,7 +73,11 @@ exit_code: 0
             "value": 0.0,
             "unit": "byte"
           },
-          "exit_code": 0
+          "exit_code": 0,
+          "time_cpu": {
+            "value": 0.0,
+            "unit": "second"
+          }
         }
       ],
       "summary": {
@@ -82,6 +90,15 @@ exit_code: 0
           "min": 0.0,
           "max": 0.0
         },
+        "time_cpu": {
+          "unit": "second",
+          "count": 2,
+          "mean": 0.0,
+          "stddev": 0.0,
+          "median": 0.0,
+          "min": 0.0,
+          "max": 0.0
+        },
         "time_system": {
           "unit": "second",
           "count": 2,
@@ -131,7 +148,11 @@ exit_code: 0
             "value": 0.0,
             "unit": "byte"
           },
-          "exit_code": 0
+          "exit_code": 0,
+          "time_cpu": {
+            "value": 0.0,
+            "unit": "second"
+          }
         },
         {
           "time_wall_clock": {
@@ -150,7 +171,11 @@ exit_code: 0
             "value": 0.0,
             "unit": "byte"
           },
-          "exit_code": 0
+          "exit_code": 0,
+          "time_cpu": {
+            "value": 0.0,
+            "unit": "second"
+          }
         }
       ],
       "summary": {
@@ -163,6 +188,15 @@ exit_code: 0
           "min": 0.0,
           "max": 0.0
         },
+        "time_cpu": {
+          "unit": "second",
+          "count": 2,
+          "mean": 0.0,
+          "stddev": 0.0,
+          "median": 0.0,
+          "min": 0.0,
+          "max": 0.0
+        },
         "time_system": {
           "unit": "second",
           "count": 2,

tests/snapshots/integration_tests__json_export_parameterized_with_reference.snap

@@ -50,7 +50,11 @@ exit_code: 0
             "value": 0.0,
             "unit": "byte"
           },
-          "exit_code": 0
+          "exit_code": 0,
+          "time_cpu": {
+            "value": 0.0,
+            "unit": "second"
+          }
         },
         {
           "time_wall_clock": {
@@ -69,7 +73,11 @@ exit_code: 0
             "value": 0.0,
             "unit": "byte"
           },
-          "exit_code": 0
+          "exit_code": 0,
+          "time_cpu": {
+            "value": 0.0,
+            "unit": "second"
+          }
         }
       ],
       "parameters": {
@@ -87,6 +95,15 @@ exit_code: 0
           "min": 0.0,
           "max": 0.0
         },
+        "time_cpu": {
+          "unit": "second",
+          "count": 2,
+          "mean": 0.0,
+          "stddev": 0.0,
+          "median": 0.0,
+          "min": 0.0,
+          "max": 0.0
+        },
         "time_system": {
           "unit": "second",
           "count": 2,
@@ -137,7 +154,11 @@ exit_code: 0
             "value": 0.0,
             "unit": "byte"
           },
-          "exit_code": 0
+          "exit_code": 0,
+          "time_cpu": {
+            "value": 0.0,
+            "unit": "second"
+          }
         },
         {
           "time_wall_clock": {
@@ -156,7 +177,11 @@ exit_code: 0
             "value": 0.0,
             "unit": "byte"
           },
-          "exit_code": 0
+          "exit_code": 0,
+          "time_cpu": {
+            "value": 0.0,
+            "unit": "second"
+          }
         }
       ],
       "parameters": {
@@ -174,6 +199,15 @@ exit_code: 0
           "min": 0.0,
           "max": 0.0
         },
+        "time_cpu": {
+          "unit": "second",
+          "count": 2,
+          "mean": 0.0,
+          "stddev": 0.0,
+          "median": 0.0,
+          "min": 0.0,
+          "max": 0.0
+        },
         "time_system": {
           "unit": "second",
           "count": 2,

tests/snapshots/integration_tests__json_export_single_run.snap

@@ -45,7 +45,11 @@ exit_code: 0
             "value": 0.0,
             "unit": "byte"
           },
-          "exit_code": 0
+          "exit_code": 0,
+          "time_cpu": {
+            "value": 0.0,
+            "unit": "second"
+          }
         }
       ],
       "summary": {
@@ -58,6 +62,15 @@ exit_code: 0
           "min": 0.0,
           "max": 0.0
         },
+        "time_cpu": {
+          "unit": "second",
+          "count": 1,
+          "mean": 0.0,
+          "stddev": null,
+          "median": 0.0,
+          "min": 0.0,
+          "max": 0.0
+        },
         "time_system": {
           "unit": "second",
           "count": 1,