Performance¶

How ai-rulez is benchmarked, how a regression is caught, and what the optimization passes changed.

Running the benchmarks¶

Every benchmark that reads a project builds a synthetic one in a temporary directory (internal/testutil.BuildBenchTree): rules, context, skills, agents and commands with frontmatter and prose, plus a git repository when the code under test asks git questions. Nothing touches the network.

task test:benchmark                 # every benchmark, small and medium
task test:benchmark:large           # adds the 2000-file project, one iteration each
go test ./internal/lint -run '^$' -bench RunFull -benchmem -cpuprofile cpu.out
go tool pprof -top -cum cpu.out

Regression tracking¶

scripts/bench.sh runs a curated subset (benchmarks that start no process and do not fsync, so they measure the code and not the machine) with -count=6 and compares the median time/op of each benchmark with scripts/bench-baseline.txt. It exits 1 when any benchmark is more than 25% slower. When benchstat is installed (or the machine is online, through go run golang.org/x/perf/cmd/benchstat@latest) its report is printed too.

scripts/bench.sh            # run, compare, fail on a >25% regression
scripts/bench.sh --update   # re-record the baseline on this machine

Time per operation depends on the CPU, so the gate only applies when the baseline was recorded on the same CPU model as the run (the cpu: line of the output). Otherwise, or when there is no baseline, it prints a warning and exits 0. The CI Benchmarks job runs the script after the full benchmark run and uploads bench-subset.txt; to make the gate bite on the CI runners, commit that artifact as scripts/bench-baseline.txt.

Optimization passes¶

Numbers are time/op and B/op on an Apple M5 Pro that was running other builds at the same time (load average near 50), taken as alternating runs of test binaries built from the commits below, so the ratios are reliable and the absolute values are not. Behavior is unchanged: the golden tests and every existing test pass, and each change that could alter a result has a test comparing it with the code it replaced.

The shell and instruction-override pre-checks compare ASCII literals without case in place and treat U+017F and U+212A as the s and k the patterns' case folding makes them, so a line the pattern would match always reaches it.

Second pass: repeated loads, publish at scale, startup¶

Same method as above, taken on a quieter machine (load average near 20): test binaries built from the commit before the pass and from the last commit of it, run back to back with -benchtime=3x -count=2 (1x for the 2000-file project), so the ratios are reliable and the absolute values are not. Output is unchanged: tests/golden passes without edits.

What changed where:

• lock --check and publish --dry-run load once and thread the result. The memo (gitutil.WithMemo) is attached to the context of one generate, drift, lock --check or publish run and dies with it; it is never process-wide, and the long-lived callers (the MCP server, the doctor, watch mode) do not get one. Questions about history (HEAD, refs, file lists) are not memoized.
• The 2.5 GB of publish --dry-run at 2000 files was mostly one allocation: the trap check read every candidate file into a fresh 1 MiB buffer. The source hash text (built whole, hashed, dropped) and the repeated configuration loads of the lock check were the next removable allocations.
• Package-level regular expressions in the four packages are lazy. The global archlint rule treats a sync.OnceValue the way it treats regexp.MustCompile (immutable in practice); the three existing OnceValue globals (in ard, tokens and generator/settings) lose one allowlisted site each.

Behavior that changed deliberately (everything else is byte-identical):

• Frontmatter is split by internal/frontmatter in the config loader (parseFrontmatter, hasUnclosedFrontmatter, FrontmatterProblem, IsOKFListing), in lint (parseDoc) and in the generator (frontmatterEnd, hash injection, listingFrontmatter, ruleFileFrontmatter, mappedFrontmatterFields). Its rules now apply to all of them: a UTF-8 BOM before the opening fence is ignored (the loader and the generator used to treat the file as having no frontmatter); a fence may carry trailing spaces or tabs, and an indented --- is not a fence (the loader and lint used to trim both sides); the empty block ---/--- is a block (the loader needed three lines); the blank line after a CRLF block is dropped from the body as it is after an LF block; ruleFileFrontmatter no longer skips blank lines before the opening fence and no longer takes ---foo for a closing fence; hash injection into a block closed on its first line works. internal/frontmatter.ClosingLine finds the closing line without building the block, so lint does not copy the YAML.
• Readers now refuse input over their limit through safefs.ReadLimited instead of passing a cut one on: an attestation file over MaxBundleBytes (AR721 at the read, not at the parse), the GitHub token and Sigstore service responses over their caps, and the publish inputs (templates, the lock, skill files, dist files and tar entries), the signing key, the LLM cache entry and secret. safefs.ReadFileLimited opens and reads with the same bound.

Third pass: the process cleanup after each spawn¶

internal/runner ends every run by killing what the command left behind. The guarantee, from the code and the tests (TestRunKillsHelpersThatLeaveTheGroup, TestRunKillsADetachedSystemBinary, TestConcurrentRunsDoNotKillEachOther): after Run returns, no process of the run is alive. That covers the process group, any descendant seen while the command ran (a helper that called setsid or setpgid is still a descendant), and a helper that detached and was reparented to init between two looks at the table, which is found by the run token every child inherits in its environment or by the write end of the marker pipe every child inherits as a descriptor (macOS withholds the environment of platform binaries, so there the descriptor is what finds them). It does not cover a helper that drops both the variable and the descriptor before detaching, and it must never touch a process of another run. On Windows the job object gives the guarantee without any table read.

The cost was in the token and descriptor search (adoptCarriers): it skips processes older than the command's root (a descendant cannot be older than its ancestor), but a Spec.ShortLived run (every git call) does not read the process table at start, so it never learned the root's start time and the search covered every process of the user, one sysctl (macOS) or /proc/<pid>/environ read (Linux) each, after every spawn.

Now a short-lived run reads the start time of its own root with a single lookup at attach (processStart: one sysctl kern.proc.pid, or one /proc/<pid>/stat), so the search covers only processes that started since the command, which is a handful. If that lookup fails the search is unbounded as before, so the guarantee is the same on every platform; nothing was weakened. The full (non-short) path already had the bound and is unchanged. Windows is unchanged.

Measured on a loaded machine (load average 16 to 28, 850 processes), go test -bench of the runner and generator packages, test binaries built before and after and run back to back. Wall times are dominated by the fork cost under that load and are not reliable; the allocations and the isolated sweep are.

The cost of the sweep now depends on how many processes start during the command, not on how many exist.

What dominates elsewhere¶

These were measured and not changed here.

• Process spawns per content file. Loading a project with 200 skills took 91 s on the loaded machine before the branch that batches git ls-files and git check-ignore per directory (0.5 s with it). The remaining wall time of generate, clean, lock and publish on 200 to 2000 files is mostly git (rev-parse, ls-files) and file-system calls; their CPU time is a small share.
• Token counting. cl100k counting of the prompt text is the largest allocator in a lint run (about a third of the bytes, in regexp2 match objects); the memo that removes it is part of the same separate branch.
• Served skills. lock --check still builds the served-skill views twice (mcp.DynamicLockChanges loads the configuration for each view and scans the skills); that code lives in internal/mcp.
• Publish at scale. What remains of publish --dry-run at 2000 files (1.36 GB over 9.9 million objects) is the strict lint run (about 60%, mostly cl100k token counting in regexp2, which the separate token memo branch removes), the lock comparison and the served views.