Performance
Halfhand is a local-first flight recorder, so its performance budget is felt,
not announced: hh replay opening instantly, hh list staying out of the way
of a shell prompt, and a long recording not falling behind the agent. This page
is the reference for the four performance-critical paths the requirements name, how to
measure them, and what the regression gate does and does not enforce.
The numbers below are reference, not the gate. The gate compares tonight's run against last night's run on the same runner class; the absolute figures shift with hardware. Treat the reference numbers as "roughly what a developer machine sees," and the gate as "did we get slower than yesterday."
The four benches
Criterion benches live in hh-core/benches/ (CLAUDE.md v1.0.0 addendum).
Four groups, one per perf-critical path:
| Group | What it measures | Target |
|---|---|---|
ingest |
EventWriter::append_event throughput — one durable round-trip through the single-writer channel per event (channel send → writer-thread INSERT → reply). |
ingest ≥5,000 events/s |
replay_index |
Store::list_event_index for 10k and 100k-event sessions — the hh replay open hot path. |
replay open <300 ms for 10k; <1 s for 100k |
blob_write |
BlobStore::put (BLAKE3 + zstd + atomic write) for 1 KiB / 1 MiB. |
file-change blob round-trips |
blob_read |
BlobStore::get (read + zstd decompress + BLAKE3 verify) for 1 KiB / 1 MiB. |
replay / inspect body fetch |
adapter_jsonl |
Claude JSONL line → event conversion via the dev-only adapter::fuzzing parser entry. |
Claude Code adapter tail |
ingest batches 128 events per criterion iteration and reports
Throughput::Elements(128), bounding the session to a small steady-state table
so append cost does not drift as an index fills. blob_write returns its temp
dir from the measured closure so the TempDir drop (the blob file + shard dir +
rmdir) happens outside the timed region — otherwise that syscall noise would
dominate and add false variance.
Run them:
just bench # all four (enables `fuzzing` for adapter_jsonl)
just bench-storage # the three storage benches, ungated (faster loop)
cargo bench -p hh-core --features fuzzing # what the nightly gate runs
The nightly gate (cargo bench then bench_compare.py) runs with bounded
criterion settings (1 s warmup, 1 s measurement, 30 samples ≈ 4 min total);
criterion's defaults (100 × 5 s ≈ 8 min per bench) would make the run ~an
hour. Tune up locally by editing bench_config() in the bench files or passing
--measurement-time / --sample-size on the CLI.
Reference numbers
From a cargo bench -p hh-core --features fuzzing run (developer-class Linux
machine, bounded settings). These are reference points, not pass/fail:
| Bench | Median | Throughput |
|---|---|---|
ingest/append_event |
18.9 ms / 128 events | ~6.8k events/s (storage layer) |
replay_index/10000 |
5.0 ms | replay open <300 ms ✓ |
replay_index/100000 |
49 ms | <1 s ✓ |
blob_write/1KiB |
6.1 ms | — |
blob_write/1MiB |
11.6 ms | ~86 MiB/s |
blob_read/1KiB |
6.7 µs | — |
blob_read/1MiB |
1.05 ms | ~950 MiB/s |
adapter_jsonl/parse_lines |
7.6 µs / 3 lines | ~395k records/s |
Ingest target (≥5,000 events/s)
The bench measures the storage layer in isolation — append_event through the
single-writer channel with synchronous = NORMAL (no per-commit fsync; the WAL
fsyncs at checkpoint, matching the "fsync on session finalize" durability design). At
that layer it sustains ~6.8k events/s, above the 5k/s target.
The ingest target is end-to-end, and the real recorder (PTY reader → adapter
parse → serialize → channel → writer) runs slower than the storage layer
alone. The synchronous = NORMAL fix lifted end-to-end ingest ~5× off the old
per-event-fsync baseline (see the ### Performance entry in the CHANGELOG);
closing the remaining gap to 5k/s end-to-end needs statement caching + a batched
writer, tracked as follow-up. The regression gate below protects the storage-
layer gain from slipping back.
Big sessions
hh replay opens by loading the event index (list_event_index), then fetches
bodies lazily through a bounded 50-entry LRU cache (ReplayData). At 100k
events the index load is 49 ms and the body cache is verified bounded at capacity
across unbounded scrolling (get_evicts_lru_to_stay_bounded_at_capacity), so
open stays <1 s and navigation does not grow memory with session size.
hh inspect --json streams a session as NDJSON through
Store::for_each_event_detail — one EventDetail at a time, never collecting
the whole session into RAM — so a 100k-event session inspects in constant
memory. (An #[ignore] test, for_each_event_detail_streams_100k_session,
exercises the 100k path end-to-end with --ignored.)
The nightly regression gate
.github/workflows/bench-nightly.yml runs the benches nightly and on
workflow_dispatch, then .github/scripts/bench_compare.py compares each
bench's median against the previous night's baseline, failing on >15%
regression. A bench is a regression when
current_median > baseline_median × 1.15.
Why a rolling baseline, not a committed one
A baseline committed to git and generated on one machine, compared on another,
is dominated by hardware differences, not code changes — a 15% gate would be
meaningless across an Intel laptop vs an ARM cloud runner. Instead the workflow
uses an actions/cache rolling baseline keyed by runner OS: each run
restores the previous run's baseline (bench-baseline-<OS>-), compares against
it, then — only on a passing run — saves its own results as the new baseline
(bench-baseline-<OS>-<run_id>). Current-vs-baseline is therefore the same
runner class, and the 15% threshold absorbs pool variance within that class.
A regressed run is never adopted as the baseline (the seed step runs only on
if: success()), so the gate always tracks the last passing run.
Establishing, not failing, on a cold cache
When the cache is cold (first run, or after eviction), there is no baseline to
compare against. Benches present in the current run but absent from the baseline
are reported as establishing — no comparison, never a failure — and the
run seeds the baseline for next time. The compare script's exit codes are part
of the contract: 0 pass, 2 regression, 3 establishing/could-not-compare.
Local comparison
just bench # run the benches (writes target/criterion)
just bench-save-baseline ./bench-baseline # snapshot this run as the baseline
# ... make a change ...
just bench # run again
just bench-compare ./bench-baseline # same script the nightly gate uses
bench-baseline/ is gitignored (a local/CI cache artifact, not committed).
bench-compare walks target/criterion/<group>/<bench>/new/estimates.json
(criterion writes this on every run — no special flag) and reads each median
point estimate in nanoseconds, criterion's native unit for estimates.json.
hh list cold start
Target: hh list cold start <50 ms on a warm cache. This is a manual check,
not CI-gated — wall-clock timing is too noisy across CI runners to gate on, and
the 50 ms budget is a developer-experience target, not a correctness contract.
The two things that used to make hh list scale with recorded history are both
fixed:
- No blob-dir scan on open.
Store::openmkdir -ps the blob dir but does not read it; blob-dir scans only happen inhh gc(prune) andhh stats(footprint), never inlist.listruns oneSELECTagainstsessions. - O(rows-needing-heal) step probe.
Store::open's step self-heal used to scan every event on every invocation, so cold start grew with total history. Migration 0002 added a partial indexidx_events_needs_heal(WHERE step IS NULL AND kind != 'terminal_output'), making the probe O(rows that actually need healing) — near-zero for a healthy store. Additive (a new index on an existing table); no breaking change.
Running the check (hyperfine)
hyperfine is a separate install (cargo install hyperfine); it is not a build
dependency. The just cold-start recipe builds a release binary, seeds a temp
data dir with a couple of sessions, warms the OS cache, and runs hh list
under hyperfine:
just cold-start
which runs (roughly):
cargo build --release --bin hh
export HH_DATA_DIR="$(mktemp -d)"
./target/release/hh run -- echo warmup >/dev/null # seed a session
hyperfine --warmup 3 --runs 20 \
"HH_DATA_DIR=$HH_DATA_DIR ./target/release/hh list"
On a warm cache (after --warmup 3), a developer-class machine sees hh list
around 2 ms — well under the 50 ms budget. The budget has headroom for the
config parse and store open (migrations are idempotent and cheap; the heal probe
is near-zero) without scaling with how much you have recorded.
If
hyperfineis unavailable, a quick fallback is a shell loop:export HH_DATA_DIR="$(mktemp -d)"; ./target/release/hh run -- echo x >/dev/null for i in $(seq 1 20); do /usr/bin/time -f '%e' ./target/release/hh list >/dev/null; done(coarser than hyperfine, but enough to spot a regression into the tens of ms).