---
name: delegate
description: Delegate bounded coding tasks to workers, with Astra or Fable coordinating scope, review, and integration. Load before dispatching coding subagents or background builds. Also use for finite model experiments, benchmarking, and worker routing decisions.
---

# Delegate — Coordinated Work and Bounded Model Experiments

Use Astra or Fable as the coordinator when available. The coordinator defines bounded tasks, selects workers, reviews their output, and owns integration and shipping. Delegate work when that improves delivery. Keep design decisions and final judgment with the coordinator.

**One implementation per task by default.** Model experiments are optional, bounded work with their own question, budget, and decision. Production work ships when its reviewed implementation and required checks pass. An experiment never holds up shipping, documentation, or issue updates.

## The First Question Is SURFACE, Not Difficulty (Chris, 2026-08-13)

Before routing anything, ask **who ends up looking at the output**. That, not how hard the ticket is, decides whether it gets delegated at all.

- **No human-visible surface, and you can verify the outcome yourself** — a unit test, a curl against the dev worker, a query against the live telemetry, a row in a store. **Delegate it whole, fire and forget, close on green.** Refactors, internal stores, access gates, migrations, CLI tooling, test plumbing, redirect rules. Run these in the background, several at once, while the interactive session does something else.
- **A UX DESIGN call with nothing decided behind it** — layout, placement, a new page or flow, an interactive surface, what a screen IS — **is NOT delegated. It is built INTERACTIVELY with Chris, against a running dev server he can drive.** His words (2026-08-13): *"anything that needs me to QA or check UX or visuals on webpages or slack interactive I'll build with you using a local server setup."* A delegated build satisfies "Chris reviews it before it lands" on paper by reporting well, and still fails this: he wants to watch the real thing while it changes. What lifts it is an artboard, not a better-written brief — the exemption is about where the DECISION was made.

**Judge by what a human ends up seeing, not by which workspace the diff is in.** A backend-only change that alters what a page shows is a UX ticket; a frontend-only change to a route's caching is not.

Why the split is drawn there (27agents, 2026-08-12): four delegated refactor tickets landed with zero QA defects, while every defect Chris found in live QA was last-mile presentation. Not one was a logic bug. Agents are reliable at logic and blind to what a person sees, because nothing in their loop renders it.

### Copy is not a blocker (Chris, 2026-08-22)

His words: *"I'm really happy for you to proceed and make small arbitrary decisions on copy. The artboard should always take precedence. I can always tweak things quickly there. If you're doing copy for agents then I'll send back evaluations who don't like the copy and you can loop on them and improve them there."*

Wording is small, reversible and cheap for him to change, so it never blocks a ticket, never forces one interactive, and never gets parked for his eyeball. Three rules, in order:

1. **The artboard always wins.** If a prototype shows the string, that string ships verbatim; copy on a prototype is real copy, not a placeholder to improve on. A worker that reworks it is overriding an approved design.
2. **Where nothing supplies the string, write it.** Pick the obvious wording, ship it, and list in the report every string you composed. Do not stop and ask. Where the surface has an artboard, add the new string to it in the same change — Chris's tweak loop runs on the canvas, so the canvas has to hold what shipped.
3. **Copy an AGENT speaks is corrected by evals, not by him.** Replies, prompts, tool descriptions, digests, result text: ship the wording, and a returned evaluation that dislikes it becomes a case you iterate against (27agents: `/eval`).

### Delegating UX — the artboard is the entry condition (Chris, 2026-08-19)

UX work is delegable when the DESIGN is already fixed somewhere the worker can look: an artboard or prototype the brief names by path (a `.dc.html`, a design canvas, a static mockup), showing layout, placement and the words. Without one it stays interactive; with one, the brief must say all four of these:

1. **The prototype is the authority; the prose is a summary of it.** Where they disagree, follow the prototype and say so in the report. `ds4-295-remove-config` had an approved `Done.dc.html` showing three periods as ONE combined bullet, read the brief's looser "one bullet per enabled period" literally, and shipped three.
2. **Follow its copy exactly, and never borrow a string from an unrelated surface** (that run filled its three bullets with sentences lifted from a settings form). Compose only what the prototype leaves out, per **Copy is not a blocker** above.
3. **Name every surface the prototype covers, and check each one.** `305-shared-settings` moved a per-bot editor into the shared card against the artboard's placement and skipped Miriam's page; `306-settings-artboards` shipped a web-only diff and missed two WORKER strings the change had made untrue. Both had artboards. Neither read all of them.
4. **Demand the ARTIFACT, not a claim about it.** "The description no longer names email" is a claim; "here is the exact string this renders, verbatim" is evidence, and a lie is then sitting in the report in plain text.

**A prototype settles the taste question — it does not defer it.** If an artboard shows the layout, whether that layout feels right is ALREADY DECIDED and the worker builds it rather than evaluates it. So **a prototyped UX ticket does not need a taste-7 worker** — the taste was spent drawing the artboard; route it on the logic. What is still open is what the prototype does not cover: a state it does not draw, an error case, a surface it never shows. Ask when that gap is a design call; write it when the gap is only wording.

## You Stay the Engineer of Record

You pick the work, write the spec-prompt, review the returned diff, run the AFFECTED gate yourself (see Which Gate below), own commit/push/CI. **Never delegate the final review or security judgment.**

Keep for yourself: anything whose design you can't fully spell out in the prompt, and security boundaries you haven't enumerated (the four-boundary tests can be delegated once YOU have named them).

**Design work turning on decisions Chris hasn't made stays interactive** (Chris, 2026-07-29). A subagent handed a design question returns a finished, defended design — and a finished design is something Chris reacts to rather than participates in, which is the wrong shape when the question is "what should this product be" rather than "what does this code do". The tell: the answer depends on his intent, not on anything discoverable in the repo. Split it instead — delegate the expensive reading (what the tickets say, where they contradict each other and the code, what each option forecloses downstream) as a **spike that stops before deciding**, then decide with him and write the doc from that conversation. Once a recommendation is on the table he is anchored, so the delegation has to end before the opinion starts. (2026-07-29: a fable agent was briefed to write the Em memory model doc and told to "be opinionated, make the recommendation and defend it" — Chris pulled it back to interactive.)

## Verify the Gate Yourself — Every Worker, Every Time

**A subagent claiming "done" is not evidence — verify the gate yourself, sonnet and opus included, not just Codex.** Confirmed 2026-07-08 (same session): two separate Agent-tool workers — one sonnet, one opus — ended their turn without ever having actually run and confirmed the gate green. One's final message was "waiting on the npm run check monitor to report back" and then it just stopped; the other exited mid-gate with no pass/fail line at all. Neither had committed. A subagent's self-report is a claim, not a receipt.

Before trusting a branch or flipping any status: confirm there IS a commit (`git log -1` on its branch/worktree), confirm the worktree is clean (`git status --short`), and re-run the gate yourself in that worktree (or after merging) rather than accepting "tests passed" prose at face value. If a subagent's final report doesn't quote the literal exit status / pass count of every gate step, treat the work as unverified until you've checked it yourself.

### Which Gate — AFFECTED, not the whole suite

**"The gate" in this skill always means the AFFECTED gate, never the full suite.** Every repo defines two, and the repo's own `CLAUDE.md` is the authority on the exact commands:

- **AFFECTED** — the tests the diff can actually reach, plus a typecheck for each area it touches. In a vitest repo that is `--changed <base-ref>`, which walks the module graph from the changed files. This is what an arm runs, and what you re-run to check the arm's claim.
- **FULL** — the whole suite, every workspace. **CI runs this on every push and is the backstop.** Neither you nor an arm runs it locally without a specific reason.

**Never put a suite-wide command in a brief** — not `npm run check`, not `npm test --workspace <name>`, not a bare `vitest run`. This is the failure mode to watch for, because it does not look like one: on 2026-08-19 three arms were each running a seven-minute worker suite simultaneously on a four-core box, load average 22.8, all three still unfinished at fifteen minutes — because each brief had spelled the gate out by hand and reached for the whole suite. The cost is not paid by the arm that runs it; it is paid by every other arm on the machine.

**"Scoped to a workspace" is not the affected gate.** A workspace's `test` script usually carries no filter, so scoping that way is the full suite wearing a smaller name — the phrase reads cheap and is not. Only an explicit `--changed` or file list narrows anything.

`scripts/worker_rules.md` carries this for the worker; your job is not to restate the gate commands in the brief, but to name the **base ref** the arm should pass to `--changed`.

**Agent-tool arms do not get those rules unless you paste them.** `codex_arm.sh` appends `worker_rules.md` to the brief and `pi_bench.py` does the same for every arm it runs — but a `sonnet`/`opus`/`fable` arm dispatched straight through the Agent tool goes through neither, so it receives no gate rule, no don't-push, no don't-delegate. **Append the contents of `scripts/worker_rules.md` to every Agent-tool brief**, the same way the scripts do. This is exactly how a hand-written brief ends up spelling the gate out from memory and reaching for the whole suite.

## Routing — Cost / Intelligence / Taste, Not Habit

Higher = better on every axis: cost is what you actually pay (higher = cheaper to run), intelligence is how hard a problem you can hand it unsupervised, taste covers UI/UX, code quality, API design, and copy.

| model          | cost | intelligence | taste | reach                                              |
|----------------|------|--------------|-------|-----------------------------------------------------|
| gpt-5.6-sol    | 7    | 8            | 5     | Codex CLI — `codex exec -c model="gpt-5.6-sol" -c model_reasoning_effort="high"` (Chris, 2026-07-29: use 5.6-sol, not 5.5; cost 7 not 9 — pricier than 5.5 was, so it is not free to run. The exact id is `gpt-5.6-sol` — bare `gpt-5.6` 400s on this account) |
| sonnet-5       | 5    | 5            | 7     | Agent tool `model: "sonnet"`                       |
| opus-5/xhigh   | 4    | 7            | 8     | Agent tool `model: "opus"` + `effort: "xhigh"` |
| fable-5        | 2    | 9            | 9     | Agent tool `model: "fable"`                        |
| gpt-6-astra    | 3    | 10           | 8     | Codex CLI — `codex exec -c model="gpt-6-astra" -c model_reasoning_effort="high"` (Chris, 2026-09-08: slightly cheaper than fable, not as good at design, one better than fable at everything else; it is the default in `~/.codex/config.toml`) |

Astra or Fable coordinates experiments with other models. A candidate can receive a bounded implementation, investigation, or review task. For production work, the coordinator reviews and verifies the result before accepting it. Use isolated frozen tasks for independent comparisons. Read `references/shadow-benchmarking.md` before starting an experiment.

**Judge total delivery cost.** Token rates alone miss execution time, coordinator review, and corrections. Read the ledger at `~/vault/data/model-bench/runs.jsonl` alongside the project delegation log. Preserve missing measurements as unknown. Compare completed outcomes under named models and harnesses. The benchmarking reference explains the limits of the summary report.

Defaults, not limits: standing permission to override without asking — if a cheaper worker's output misses the bar, redo it on a smarter one. Judge the output, not the price tag; escalating costs less than shipping mediocre work. Cost is a tie-breaker only; when the axes conflict on anything that ships, **intelligence > taste > cost**. **Never use Haiku.**

Work whose judgment is *aesthetic* needs taste ≥ 7: visual design, or an API shape other people will have to live with. Read that narrowly — a ticket that merely composes a string does not qualify, because copy is correctable after it ships (see **Copy is not a blocker**). **An artboard removes the aesthetic judgment from the ticket, so the taste floor does not apply to it** (Chris, 2026-08-19) — what is left is implementing what it shows. Route those on the logic, not the taste column. See **Delegating UX** above. Logic a user can reach but which makes no copy or design call — a redirect rule, an access gate, a migration, a record store, a CLI — is not a taste job. Read "user-facing" broadly and it swallows nearly every ticket in a product repo, which quietly disqualifies the cheap workers from everything.

**Who builds what** (cheapest worker that clears the bar wins):

- **Codex / gpt-5.6-sol** (`codex exec`, see below) — **the default worker, and it should be carrying a real share of the load.** Two defaults, two scopes: `scripts/codex_arm.sh` pins `gpt-5.6-sol` for worker runs, and `~/.codex/config.toml` sets `gpt-6-astra` as the CLI default for everything else, so pass the model explicitly when you want Astra as a worker. Chris's own framing (2026-08-03): it lands between opus and fable on intelligence, and cheaper than either. That is what the table's 8 means — reach for it on any well-specified slice on a STABLE spec that makes no copy or design call: clear-spec implementation, internal stores and APIs, CLI tooling, data analysis, migrations. Also root-cause debugging where the fix needs researching an external system's real behavior. Fire-and-forget — no mid-flight correction — and graded adversarially for its one failure mode neither Agent-tool worker has shown: weakening a test to make it pass rather than fixing the code (the review step below exists to catch it). That failure mode is also the one place to route around it: the tests guarding a security or privacy boundary are exactly where a quietly weakened assertion costs most, so either put those on opus/xhigh or diff every touched test file against its pre-change version before accepting.
- **`model: "sonnet"` / `"opus"`** (Agent tool) — when you need to steer mid-flight (SendMessage a scope correction while it runs): sonnet for mechanical work (bulk edits, renames, test scaffolds, doc updates, broad searches), opus/xhigh for well-specified implementation slices and security-boundary work. Request xhigh on every opus dispatch — pass `effort: "xhigh"` (or state it in the brief where the surface doesn't expose it) and record `opus/xhigh` in the log; the default can silently drop to a lower tier (session effort, cron runs, harness changes), and the table's intelligence rating assumes xhigh.
- **`model: "fable"`** (Agent tool) — top intelligence and top taste: the hardest well-specified slices and any surface where the judgment is aesthetic (taste ≥ 7). The premium option; reserve it for work that earns it.

**Keep dispatch simple.** `scripts/codex_arm.sh <label> <brief-file> [base-ref]` prepares a worktree and starts a Codex worker. Use the current tool surface for other workers. Check dependency setup before dispatch instead of assuming that a launcher supplies it.

## Bounded model experiments

Astra or Fable owns the experiment question, task boundaries, review, and final decision. The question is which tasks a model can handle usefully under that supervision. A worker does not need to outperform the coordinator to earn work.

Define the task or finite task batch, candidate model and harness, acceptance criteria, time and cost limits, and decision before dispatch. Include coordinator review and corrections in the budget. A small initial batch of 5–10 tasks is a screening option, not a required sample size.

Use one of these forms:

- **Supervised task:** give a worker a bounded piece of useful work, then review and verify its result. Record help and corrections.
- **Focused review or investigation:** ask a specific question with a concrete deliverable. Record whether its finding changed the implementation or decision.
- **Independent comparison:** give candidates the same frozen task and starting source in isolated environments. The comparison can run after production work ships.

Read `references/shadow-benchmarking.md` for isolation, launchers, measurement, and grading. Keep experiments off the production dependency chain. At the budget limit, record the result and decide whether to use the model for that task class, revise the experiment, or stop.

## Task Shape — Plan, Fix, Spike

Name the task shape before dispatch. Workers behave better when the brief names the mindset:

- **Plan/build** — new behaviour from a stable spec. Worker writes red tests and code together, then lands a clean implementation slice.
- **Fix** — existing behaviour is wrong. Existing tests must stay green and unchanged unless the defect lacks coverage; add a failing regression test first, then fix the code. Watch especially for workers weakening a test to make their patch pass.
- **Spike** — read-only exploration. No production code, no reusable implementation, no opportunistic cleanup. The output is findings, options, and a proposed next spec/ticket.

If a run starts as a spike and discovers the build path, stop and write the follow-on spec/ticket. Don't let exploratory context turn into production code by drift.

## Briefing — Every Worker Starts Cold

Point the worker at the canonical spec document — a ticket file, a design doc — in its own worktree and let it read the file. Don't write a parallel plan.md or restate the spec in prose. Add only what the spec doesn't already say: task shape (plan/fix/spike), the project's testing discipline (red-first), the relevant CLAUDE.md gotchas verbatim (a cold worker doesn't have your session's context), whether other parallel builds are running (demand a surgical diff if so), the gate command, the commit message format, and "do not push". If the repo has `architecture.md`, `ARCHITECTURE.md`, or ADRs, name them in the brief and require the worker/reviewer to check fit against them. Same spec, same brief, different worker — that's what keeps runs genuinely comparable.

### Challenger coaching and review

`pi_bench.py` owns challenger coaching and the post-stop review pass. Before using or grading a challenger, read `references/shadow-benchmarking.md`; do not copy its coaching block into a brief.

## Isolation and Landing

**Every delegated build runs in its OWN git worktree, even a single sequential task** (Chris, 2026-07-07): pass `isolation: "worktree"` to the Agent tool; for Codex, `scripts/codex_arm.sh <label> <brief-file> [base-ref]`, which prepares the worktree and starts the run.

**Prepare dependencies in the worker environment.** Check the launcher and project instructions. Use a private install or copy where tools can modify dependencies. Shared writable symlinks let a worker damage another checkout. State the setup in the brief. A worker never edits the main working tree directly. After integration, run the affected gate against the merged change. Fix a failing guard instead of weakening it.

**Worktrees are for CODE. A worker only writing tickets or docs doesn't need one** (Chris, 2026-07-29) — no gate to run, no diff to review, nothing to conflict over, so the isolation buys nothing. Let it write in the main tree. **What that costs you is `git add -A`**: while any worker is in flight, commit explicit paths only, or a doc agent's in-progress file rides into an unrelated commit and is invisible until someone reads that commit later.

**Never `git merge --no-ff` a worktree branch into main (Chris, 2026-07-09).** Landing is always either a REBASE (rebase the worktree branch onto current main, then `git merge --ff-only`) or a SQUASH MERGE (`git merge --squash <branch>` then one commit). Both keep history linear, one real commit per landed change. If a worktree branch has several of its own commits (a build commit plus a hardening follow-up), squash them into the one landing commit rather than carrying the worktree's internal history onto main.

**Land from the MAIN checkout, never from inside the worktree (2026-08-24, four times in one session).** The rebase belongs in the worktree, because that is where the branch lives. The merge and the push do NOT. Run `git merge --ff-only <branch>` inside the worktree and git answers `Already up to date`, because HEAD there IS that branch; the following `git push origin main` then pushes the main tree's unchanged `main` ref and prints success. Nothing has landed, both commands look like they worked, and the only tell is a `git log` in the other directory. It happened on four separate tickets in one day to a coordinator who spotted it every time and repeated it anyway, because the rebase genuinely has to happen in the worktree and the hand carries on. Sequence: rebase in the worktree, `cd` back to the main checkout, merge and push there. **If a `--ff-only` merge says `Already up to date` for a branch you know is ahead, you are in the wrong directory.**
**Cleanup is part of accepting a worker, before the next dispatch or handoff.** Once its change is reviewed, landed and the affected gate passes, preserve the commit/ref, report and required regression evidence outside the worktree. Confirm its processes have finished, then remove that owned worktree with `git worktree remove`; use `git branch -d` only for merged branches. Keep unmerged candidate refs and unresolved work. If cleanup must wait, record the reason and owner instead of silently accumulating directories. Workers leave cleanup to the coordinator.

Read-only investigations need no dependency copy. Coding workers need private writable dependencies; reuse download caches or supported copy-on-write clones. A completed worker's private dependency directory can transfer to one successor when package manifests, lockfile, runtime/platform and install configuration match; never share it between active workers. Otherwise remove reproducible dependencies/build caches after acceptance. `--reflink=auto` may silently make a full copy on unsupported filesystems. On9September2026,19 idle Storybook worktrees retained roughly12GB of dependencies; cleanup belongs to each task, not a later disk emergency.

**A SHADOW's worktree is spent once it has been GRADED, not once it lands — because it never lands.** The rule above says "after landing", and a shadow never lands, so it needs its own trigger. On 2026-08-07 a repo held **33 worktrees, 24 of them shadows**, going back three weeks. The bench ledger is the record of a shadow run; the working directory is not. So the lifecycle is: arm runs → arm is graded into `runs.jsonl` → worktree goes.

**Sweep with the script, not by hand** — the same delete-the-setup logic that gave codex and the shadow their own one-command scripts. Cleaning up manually is exactly the friction that gets skipped:

```bash
scripts/sweep_worktrees.sh                 # dry run, shows what would go
scripts/sweep_worktrees.sh --yes           # do it
scripts/compress_logs.sh --yes             # gzip finished bench logs (dry run without --yes)
```

Bench logs are evidence, so they stay, but they stay gzipped. `compress_logs.sh` gzips every log in `~/vault/data/model-bench/logs/` that has been idle for a day and is not held open. On 2026-09-09 that directory held 35GB of plain JSONL and filled the disk. Run it with the sweep.

The sweep uses sentinels and commit age, which do not establish ownership or completion across sessions. Review its dry run before removing worktrees. It removes worktrees freely but only deletes **merged** branches — `git branch -d` refuses unmerged ones, so a graded shadow's diff survives as a ref, which costs nothing and keeps the evidence readable.

Before cleanup, read `references/arm-safety.md` and identify the worktrees this session owns. Clean up only those runs after their result and evidence are recorded.

**For parallel fleets, treat landing as its own step.** Keep divergence windows short: rebase/land completed branches frequently rather than letting 6–12 independent worktrees age against trunk. Enable and use `git rerere` where the repo has recurring conflict patterns; it records the conflict-resolution recipe and replays it when the same shape returns. Rerere is assistance, not authority: inspect the resolved diff, then rerun the AFFECTED gate after each merge before landing the next branch.

## Running Codex

**Run Astra (gpt-6-astra) through `codex exec --yolo`, never through pi** (Chris, 2026-09-08). The Codex harness is tied to the model and gets more out of it than pi's openai-codex provider does. The same goes for every Codex model. Background a self-contained run against the worktree:

```bash
timeout 7200 codex exec --disable plugins --disable plugin_sharing \
  --yolo -C <worktree-dir> \
  -c model="gpt-5.6-sol" -c model_reasoning_effort="high" - < brief.txt
```

The trailing `-` reads the prompt from stdin (a scratchpad brief file — NOT a repo plan.md); `--disable plugins --disable plugin_sharing` keeps autonomous worker runs on the narrow code-editing surface even if the local Codex install has marketplace plugins enabled; `--yolo` grants danger-full-access + approval=never (codex 0.153); `-C <dir>` sets the working root. **2-hour cap** — raised from 40 minutes (Chris, 2026-08-19), which a real multi-part ticket runs straight past. The cap is a guard against a runaway, not a budget: a run that hits it exits 124 and can leave unfinished work. Record a timeout as incomplete. For production work, recover the task as needed. For an experiment, stop at its declared budget and record a resumed attempt separately.

**Run it as exactly that shape: one command, literal absolute paths, `run_in_background: true`.** The allow rule `Bash(codex exec *)` in `~/.claude/settings.json` is what lets this run without a permission prompt, and it matches each shell segment on its own. It strips a leading `timeout` (so no separate `timeout` rule is needed), but it cannot see past a variable assignment (`S=/tmp/...;`), a `$VAR` in a redirect target, a `cd ... &&` prefix, or a trailing `; echo "exit=$?" >> log`. Every one of those made the whole command fall to the classifier and prompt Chris (2026-09-09). Write the scratchpad path out in full for the brief and the log, and take the exit code from the background task notification instead of an `echo`.

**`codex exec resume` has a DIFFERENT flag surface — do NOT copy the exec flags onto it:** it takes `[SESSION_ID] [PROMPT]` plus `-c`/`--last`/`--all`, but NOT `-C` and NOT `--yolo` (passing either errors instantly), and it FILTERS sessions by cwd (so `--last` from a different dir finds nothing — run from the session's cwd, pass the SESSION_ID, or add `--all`). **For a review-driven fix, prefer a FRESH `codex exec`** with a self-contained brief (name the prior commit + files + exact changes) over `resume` — same result, none of the resume quirks; reserve `resume` for finishing a genuine mid-task timeout (exit 124, dirty tree).

## Review Before Merge

**Review the surrounding surface, not just the diff.** Delegated work reliably nails the spec but misses ADJACENT copy its change falsifies (a hardcoded default beside the new form, a docstring now wrong, a test it edited to match its own regression). Read outward from the diff before accepting — for Codex specifically, diff every touched test file against its pre-change version, not just skim for green.

**Independent review + smoke pass before merge is cheap insurance for every worker, not just Codex** — a fresh subagent (fable or opus, optionally a Codex pass as an independent third perspective) adversarially reviews the diff against the spec and the repo's architecture/ADR context (WebFetch to confirm any external-system claim), and a second subagent actually RUNS the result (dev server, real requests, captured traffic) rather than reading code. This is what caught a REJECT-worthy break in an opus-built dashboard change as readily as in a codex one.

If the repo has a mutation-test command and the touched area is high-risk, run it before accepting agent-written tests. This is a targeted anti-gaming gate: mutation testing is too expensive for every slice, but valuable where a worker may have written tests that bless its own implementation rather than prove the behaviour.

## Continuation and Parallelism

**Prefer continuing an idle agent** (SendMessage) over a cold spawn for small follow-on work in the same area — it already carries the repo context. Watch for idle-wait stalls; a short nudge recovers them. Codex has no equivalent mid-run steering; a correction means waiting for it to exit and resuming.

Parallelize independent tasks as simultaneous worktree runs across any mix of workers; serialize only when two tasks must edit the same seam and the merge would be a rewrite rather than a resolution.

## Delegation Log

If the project keeps a delegation log (see its CLAUDE.md), append one entry per delegation — task, WORKER (opus/xhigh, sonnet, fable, codex), outcome, rework — failures as faithfully as wins. This is what makes a worker comparison honest instead of anecdotal.

## Integration

# Arm safety

Read this reference before changing a Delegate launcher script or sweeping worktrees.

## Never edit a script while it is running

Bash reads a script incrementally from disk, by byte offset, as it executes. Rewriting the file in place under a running interpreter moves the ground beneath it: at the next command boundary it reads from the old offset into new content and dies, usually silently.

On 2026-08-19 this killed a codex arm 38 minutes into a large ticket. Chris asked for the 40-minute cap to be raised, the edit went straight into `codex_arm.sh` while an arm was executing it, and the script died before reaching `arm_done`. The diagnosis is available without `dmesg`: **the sentinel is written after the timeout regardless of exit status, so a missing sentinel means the script never got there** — the run did not time out, it was destroyed.

The damage is worse than the lost run, because two of the rig's own safety nets die with it:

- **no sentinel**, so an `arm_wait` armed on that branch waits forever — the same silent-watch failure the sentinel exists to prevent, arriving by a new route
- **no ledger row**, so a paid arm vanishes with no record, and the pair's comparison quietly becomes one-sided in the challenger's favour

So: before editing any script in `scripts/`, check whether an arm is running it (`ps -eo args | grep '[c]odex_arm'`). If one is, either wait, or write the new version to a temp file and `mv` it into place — a rename is atomic and swaps the inode, leaving the running process reading its original file to the end.

And when an arm ends with no sentinel, do not grade it. Recover the work in place with a fresh `codex exec` pointed at the dirty worktree, and record the arm as inconclusive: whatever it says about the model is unmeasured, because the run never finished on its own terms.

## The sweep is not multi-session safe

`sweep_worktrees.sh` decides a worktree is spent from its sentinel and the age of
its tip. Both are properties of the ARM, not of the session that launched it — so
in a checkout shared by two sessions it will happily remove the other session's
worktrees. That happened on 2026-08-19: a sweep at shutdown took `ds4-fix-300`,
which belonged to a session working a different ticket in the same repo.

Nothing was destroyed, because the safety that matters held: it removes
worktrees freely but only deletes MERGED branches, so an unmerged arm survives as
a ref and `git worktree add` brings it back. Treat that as the reason it was
recoverable, not as a reason it was fine.

Before sweeping in a repo where another session may be working: `git worktree
list` and account for each one. Sweep the arms you launched by name, and leave
the rest — "spent" is a judgement only the session that dispatched an arm can
make.

## An arm can wedge ITSELF, and every existing guard reads that as slow work

On 2026-08-22 a DeepSeek shadow on 27agents #263 sat for 76 minutes having written nothing. It had authored a throwaway Python script to measure word counts and left `k` un-incremented:

```python
while k < len(src) and not (src[k].startswith('    schema:') and k > j-1):
    pass
```

Its own bash tool call could never return, so the model was not stuck reasoning: it was blocked on a child burning a full core.

**Nothing in the rig could see this.** The sentinel is absent, exactly as it is for an arm still working. `pgrep` finds the arm alive, because it is. A liveness check watching for process DEATH reports healthy, because the process is healthier than it should be. The clean worktree reads as an audit-heavy ticket that has not started writing yet, which is precisely what this one looked like. Every signal agreed on "still going".

**The one signal that separates them is log growth.** `pi_bench.py` streams events to `~/vault/data/model-bench/logs/<stamp>-<label>.jsonl`. A working arm grows that file continuously: this one added 640KB in the first 60 seconds after being unwedged, and exactly 0 bytes in the 24 minutes before. So a watch on an arm should check three things, not two: the sentinel, the process, and whether the log has grown. Five flat minutes on a live process is a stall worth looking at.

**Diagnose with the process tree, not the log.** `ps -eo pid,ppid,etimes,stat,args --forest` under the `pi_bench.py` pid shows the whole chain, and a child in state `R` with an elapsed time close to the stall's duration is the culprit, with its full command line printed beside it. That is where the offending script is readable.

**Unwedging beats killing, and it must be recorded.** Kill the runaway child **by PID** (never `pkill -f`, which matches your own shell — see the note above), and the blocked tool call returns and the model carries on. That is a repair to the environment rather than steering the model, but it is still an intervention: the arm's wall-clock and cost are no longer comparable to its control, so say so in the ledger row and do not publish its timing. Grade the diff if one arrives; treat the timing as void.

The general shape is the one this file keeps rediscovering: **silence is not success, and a guard that only knows how to detect death cannot detect a hang.**

## A clean worktree is not an idle arm: read `git log`, never `git status`

Ten minutes after writing the section above, the same watch on the same arm produced the opposite error, and it was worse. The arm had committed at 23:53. It was killed at 00:20 for having "produced nothing".

The whole mistake is one command. `git status --short | wc -l` returns **0** for an arm that has committed its work, and **0** for an arm that has done nothing at all. Those are the two states you most need to tell apart, and the dirty count renders them identically. The kill decision, and a monitor's own deadline message, were both built on it.

The reason it survives casual review is that it reads correctly EARLY: an arm mid-flight has a dirty tree, so the count is a fine progress signal right up to the moment the arm succeeds, at which point it silently inverts. The signal is at its most convincing exactly when it is wrong.

**Read progress from commits, and fall back to dirtiness only as a secondary.**

```bash
git -C "$W" log --oneline "$BASE"..HEAD     # the real answer
git -C "$W" status --short | wc -l          # only meaningful ALONGSIDE the above
```

The sentinel already carries both facts (`head=` and `uncommitted_files=`) and it was right the whole time. It said `head=da7490c` with a real commit subject. **When a sentinel and your own poll disagree, the sentinel is the one that was written by the arm.**

Cost of not knowing this: a successful arm killed, and a hand-written ledger row asserting it produced nothing, which had to be corrected. A wrong row is worse than a missing one, because the ledger is the only record anyone reads later.

# Bounded model experiments

Read this reference before running or grading a model experiment. The parent Delegate skill owns production task choice, verification, and shipping.

## Coordinator and purpose

Use Astra or Fable as the coordinator when available. The coordinator selects a bounded task, writes its acceptance criteria, reviews the result, and makes the routing decision. Other models can earn useful work without matching the coordinator across all tasks.

Choose the experiment that answers the question:

- **Supervised task:** can this worker complete a bounded implementation under the coordinator's review? Its reviewed work can ship through the normal delivery process.
- **Focused review or investigation:** does this worker find useful omissions, defects, or evidence within a named scope?
- **Independent comparison:** how do candidate implementations compare on the same frozen task? A shadow is an experimental implementation that does not ship directly.

Record the form explicitly. Supervised results measure the worker and coordinator together. They do not establish independent model capability.

## Bound the experiment before dispatch

Record a short experiment brief alongside the ledger artifacts. Include:

- The question and task or finite task list.
- The coordinator, candidate model, harness, and starting source revision.
- The deliverable and observable acceptance criteria.
- The time and cost budget, including review passes and coordinator effort.
- The help permitted during the run and how interventions are recorded.
- The decision at the end: use the model for a named task class, revise the experiment, or stop.

A batch of 5–10 tasks can screen a candidate. It is not a statistical reliability claim or a mandatory batch size. One bounded task is enough for a specific investigation.

Set a concrete deadline before launch. Use the runner's timeout when available and monitor the whole process, including review passes. The launchers do not enforce a campaign budget. The coordinator owns that limit. At the limit, stop the run and record unfinished work as incomplete. Record a resumed run as another attempt with its own budget.

Production work ships when its reviewed implementation and required checks pass. Experiments never hold up merges, documentation, or issue updates. If an experiment competes with production tests for machine capacity, pause or stop the experiment. Count test processes as well as model workers when setting concurrency.

## Isolate independent comparisons

Freeze the task text, attachments, and starting source before dispatch. Give each candidate the same acceptance criteria. Record harness coaching and permitted help separately.

Use a separate repository containing only the starting source and frozen task inputs. A worktree in the production repository shares its git object database and can expose the completed answer. Simultaneous launch does not prevent that exposure.

Restrict the experiment environment so it cannot read the production checkout, control branches, later issue comments, or completed answers. A separate directory alone does not enforce this restriction. Use private dependencies or read-only mounts instead of shared writable dependency symlinks. The existing launchers prepare worktrees but do not enforce these access restrictions.

If the available tools cannot provide that isolation, run a supervised task or record the comparison as non-independent. Do not preserve independence by making production wait.

If an answer becomes accessible during a run, record the exposure. Compare matching file blobs and commit messages when investigating possible copying. Matching artifacts are a reason to inspect access and logs, not proof of copying by themselves.

## Existing launchers and measurements

Keep the current model runners, raw logs, coaching, review passes, and ledger. Use the model's supported harness. Codex models use Codex as specified in the parent skill. A pi-compatible candidate can use the benchmark runner.

From the isolated experiment repository, the launcher interface is:

```bash
scripts/shadow_arm.sh <label> <brief-file> <base-ref> [model]
```

Resolve `scripts/` from the Delegate skill directory. Pass the frozen base explicitly. Select the model deliberately rather than treating the script's default as an active campaign.

`shadow_arm.sh` calls `pi_bench.py`. The runner writes raw logs under `~/vault/data/model-bench/logs/` and measurements to `~/vault/data/model-bench/runs.jsonl`. Finished logs get gzipped by `compress_logs.sh`, so the `log` path in the ledger can carry a `.gz` suffix on disk. Read one with `gzip -dc <path>.gz`. `PRICES` in the runner owns the configured token rates. Label these values as token-cost estimates, not provider invoices. Pi's own price fallback can be wrong for model tags absent from its catalog.

`codex_arm.sh` records its control measurements. For Agent-tool runs, capture duration and usage when the completion notification arrives. Where a compatible transcript exists, the recorder supports:

```bash
python3 scripts/pi_bench.py --record-control \
  --label <comparison-label> --model <model> \
  --from-agent-transcript <transcript-path>
```

Use the same comparison label for paired candidates. Give retries distinct attempt labels and link them in the experiment note. Preserve unknown cost or duration as null. Never estimate one model's usage from another's.

**Validate the comparison report before using its totals.** The current `--compare` implementation converts missing costs to zero and can crash on division by zero. It also replaces measured rows with later grade-only rows. Use the raw ledger and logs for a checked analysis until those paths are corrected. Preserve measurements when resolving later annotations. Require comparable completed candidates for speed or cost ratios.

Record coordinator review time, corrections, adopted findings, and actual production delay alongside the model measurements. Experimental runtime is not the same as development time lost. Costs of two implementations are additive unless one replaces the other.

## Coaching and review passes

`pi_bench.py` owns the coaching text, `COACHING_VERSION`, review prompt, and `MAX_REVIEW_PASSES`. Do not copy those prompts into the brief. Include review passes in the experiment budget.

The runner can start a fresh-context review after the model's initial stop. A commit does not mean the process is finished. Wait for process completion before grading the final experimental artifact. Production can ship independently.

Read `passes`, `review_passes`, `review_verdict`, and `review_stop_reason` alongside the diff. Usage and duration cover all passes. `stop_reason` describes the first pass. Record the model and harness version together when comparing runs.

## Monitor, stop, and clean up

Read [arm-safety.md](arm-safety.md) before launcher changes or cleanup. Preserve these operational checks:

- Use completion sentinels and the actual process exit. A log phrase or `pgrep` self-match can leave a watcher running forever.
- Verify the sentinel directory and run identity. A stale sentinel from an interrupted attempt does not describe its replacement.
- Check log growth and the process tree when progress stops. A live process can be blocked on a runaway child.
- Read commits since the starting revision alongside the dirty-file count. A clean worktree can contain completed work.
- Stop only the processes owned by this experiment. Record interventions, exit status, and the stop reason.
- Preserve the diff or commit, raw logs, and grade before removing this experiment's worktree. Leave other sessions' worktrees alone.
- Gzip finished logs with `compress_logs.sh` once no arm is writing them. A single long arm can write 3GB of plain JSONL.

## Grade and decide

The coordinator reads the diff against the task and verifies the affected checks. Grade capability by locating its implementation, not by searching for the control's identifiers. Inspect changed tests for weakened assertions. A green gate alone does not establish that every requirement was completed.

Record measurements when the run ends. Record the grade after review. Preserve both under the same run identity. Use the existing ledger vocabulary consistently:

- `shipped-clean`: the coordinator judged the completed implementation landable without fixes.
- `shipped-with-fixes`: the completed implementation needed corrections before acceptance.
- `rejected`: the completed implementation failed acceptance, including weakened tests.
- `inconclusive`: interruption, missing evidence, or answer exposure prevents an independent quality conclusion.

For shadows, the `shipped` labels describe landability, not deployment. For reviews and investigations, record whether the requested deliverable was completed and useful. Do not invent an implementation grade for a read-only task.

For independent pairs, record `shadow_vs_control` as `SHADOW-BETTER`, `near-equal`, or `control-better`, with the concrete reason. Record inconclusive pairs explicitly. Keep first-attempt results separate from results after coordinator corrections.

Write what the worker found, what it missed, what help it needed, and whether the finding was adopted. Distinguish a weak brief, environment failure, harness limitation, and model error. Check surprising ratios against raw usage before reporting them.

At the experiment's end, make the declared decision and report it in the current conversation or existing authorized reporting channel. Name the task class that earned or lost confidence. A useful scoped worker can earn more bounded tasks without becoming the default coordinator or gaining automatic approval to ship.

#!/usr/bin/env bash
# Shared helpers for the build arms (codex_arm.sh, shadow_arm.sh).
# Sourced, never executed.

# The status dir depends on $root, which the ARM SCRIPTS set but a bare
# `source arm_common.sh` does not. Unset, the helpers below resolved
# "/.claude/arm-status", a path that exists on some boxes and never receives a
# sentinel — so arm_wait polled a directory that could not answer and spun
# silently. On 2026-08-11 that ate 8 hours: both arms of ticket 242 finished in
# 20 minutes and the watch never fired. Default it here so sourcing the file
# standalone (the documented way to run arm_wait as a monitor) just works.
: "${root:=$(git rev-parse --show-toplevel 2>/dev/null || pwd)}"

# Record that an arm has finished, as a FILE.
#
# Why a file and not a process check: the obvious way to wait on a backgrounded
# arm is `pgrep -f '<something in its command line>'`, and that is a trap. pgrep
# matches full command lines, including the WATCHER'S OWN — so a poll loop
# containing the pattern literally matches itself, the arm reads as running
# forever, and the watch never fires. On 2026-08-04 that swallowed the
# completion of two arms that had both finished cleanly, after the same
# self-match had already been diagnosed twice that day. The `[c]odex` bracket
# workaround relies on remembering it in the moment, which is exactly what
# failed. A sentinel cannot match itself.
#
# Usage: arm_done <branch> <exit-status> <worktree>
arm_done() {
  local branch=$1 status=$2 wt=$3
  local dir="${ARM_STATUS_DIR:-$root/.claude/arm-status}"
  mkdir -p "$dir"
  local head
  head=$(git -C "$wt" log --oneline -1 2>/dev/null || echo "NO COMMIT")
  local dirty
  dirty=$(git -C "$wt" status --porcelain 2>/dev/null | wc -l)
  # Written in one move so a reader never sees a half-file.
  cat > "$dir/.$branch.done.tmp" <<EOF
branch=$branch
exit=$status
head=$head
uncommitted_files=$dirty
worktree=$wt
EOF
  mv "$dir/.$branch.done.tmp" "$dir/$branch.done"
}

# Wait for arms to finish and print each one's sentinel as it lands. Give it the
# branch names; it exits when all of them are done. Safe to run as a monitor:
# it watches files, so nothing it prints can be confused for a live process.
#
# Usage: arm_wait <branch> [branch...]
arm_wait() {
  local dir="${ARM_STATUS_DIR:-$root/.claude/arm-status}"
  local remaining=("$@")
  while [ ${#remaining[@]} -gt 0 ]; do
    local still=()
    local b
    for b in "${remaining[@]}"; do
      if [ -f "$dir/$b.done" ]; then
        echo "ARM FINISHED: $b"
        cat "$dir/$b.done"
      else
        still+=("$b")
      fi
    done
    remaining=("${still[@]}")
    [ ${#remaining[@]} -eq 0 ] && break
    sleep 20
  done
}

#!/usr/bin/env bash
# Prepare a BLINDED comparison packet for two arms of the same ticket.
#
# Every grade in the bench ledger was written by the same agent that launched the
# arms, wrote both briefs, and had already formed a view about which model should
# win. That is the weakest link in the whole experiment: the three "the shadow
# beat the control" entries rest on the coordinator's own word, and on 2026-08-06
# a worker's confident comparison of two branches turned out to describe a
# different arm than the one it named. Grading is the step most improved by not
# knowing whose work you are looking at.
#
# This script does the part that must be deterministic — extract both diffs,
# shuffle them, and put the answer somewhere the grader cannot see — and stops.
# Dispatching the grader and reading its verdict stays a judgement call.
#
# Usage:
#   blind_grade.sh <base-ref> <branch-a> <branch-b> [out-dir]
#
# Then hand <out-dir>/PROMPT.md to a FRESH agent that has not seen either run,
# and only afterwards read <out-dir>/key.txt to unblind.
set -euo pipefail

base=${1:?usage: blind_grade.sh <base-ref> <branch-a> <branch-b> [out-dir]}
a=${2:?usage: blind_grade.sh <base-ref> <branch-a> <branch-b> [out-dir]}
b=${3:?usage: blind_grade.sh <base-ref> <branch-a> <branch-b> [out-dir]}
root=$(git rev-parse --show-toplevel)
out=${4:-$root/.claude/blind-grade/$(date +%Y%m%dT%H%M%S)}
mkdir -p "$out"

# Coin flip, so the coordinator cannot infer the mapping from the argument order
# it typed. Arm 1 is not "the first one I passed" half the time.
if [ $((RANDOM % 2)) -eq 0 ]; then arm1=$a; arm2=$b; else arm1=$b; arm2=$a; fi

arms=("$arm1" "$arm2")
for n in 1 2; do
  # A plain diff carries file paths and content but no branch name or commit
  # message, so it is already close to anonymous — the identity leaks that remain
  # are in the content, which is what the warning below is for.
  git -C "$root" diff "$base".."${arms[n-1]}" > "$out/arm$n.diff"
done

printf 'arm1=%s\narm2=%s\nbase=%s\n' "$arm1" "$arm2" "$base" > "$out/key.txt"

# The two files that would unblind the packet instantly, and which the worker
# rules forbid an arm from touching at all. If one shows up here, the run broke
# its boundary AND the comparison is compromised; say so loudly rather than
# quietly grading a contaminated packet.
if grep -lE 'model-delegation-experiment\.md|model-bench/runs\.jsonl' "$out"/arm[12].diff >/dev/null 2>&1; then
  echo "blind_grade: WARNING an arm touched the delegation log or bench ledger." >&2
  echo "blind_grade: that breaks worker_rules.md and identifies the arm. Investigate before grading." >&2
fi

cat > "$out/PROMPT.md" <<EOF
Two agents independently implemented the same ticket from the same starting
commit and the same brief. You are grading them blind: you are not told which
model wrote which, and you should not try to guess.

- \`$out/arm1.diff\`
- \`$out/arm2.diff\`

Read both diffs in full, against the ticket they were given.

Answer, in this order:
1. **Which would you merge, and why?** Name one. "Both are fine" is not an
   answer; if they are genuinely equivalent, say which you would merge on a
   coin-flip basis and what would have separated them.
2. **What would you have to fix first** in the one you picked?
3. **Did either weaken, delete, or narrow an existing test** to make its change
   pass? Quote it if so.
4. **Did either accept a false premise** in the brief rather than checking it
   against the code?
5. **Can you tell which is the more capable model?** If yes, say what gave it
   away. If no, say so plainly — that is a real and useful result.

Judge the diffs, not the commit hygiene. Do not run the test suite; assume both
report a green gate, because both did.
EOF

echo "blind_grade: packet at $out"
echo "blind_grade: hand $out/PROMPT.md to a fresh agent; key.txt holds the mapping."

#!/usr/bin/env bash
# Launch a Codex build arm in one command.
#
# It exists for the same reason shadow_arm.sh does: an Agent-tool arm is a single
# call with isolation:"worktree", while codex was a manual worktree, a manual
# dependency symlink and a five-flag invocation. Routing follows the path of
# least setup, and on 2026-08-03 that asymmetry produced a ten-ticket raft with
# zero codex runs in it. One command each, so the choice is about fit.
#
# Usage:
#   codex_arm.sh <label> <brief-file> [base-ref] [model]
#
#   label      short slug, e.g. 218-tool-round (also names the branch/worktree)
#   brief-file the brief to hand the worker, read on stdin by codex exec
#   base-ref   commit to branch from (default: HEAD)
#   model      codex model id (default: gpt-5.6-sol — bare gpt-5.6 400s on this
#              account, so pass the full id if you override)
#
# Run it from anywhere inside the repo. Backgrounding is the caller's job.
set -euo pipefail

label=${1:?usage: codex_arm.sh <label> <brief-file> [base-ref] [model]}
brief=${2:?usage: codex_arm.sh <label> <brief-file> [base-ref] [model]}
base=${3:-HEAD}
model=${4:-gpt-5.6-sol}

[ -f "$brief" ] || { echo "codex_arm: no such brief: $brief" >&2; exit 2; }

root=$(git rev-parse --show-toplevel)
# shellcheck source=arm_common.sh
. "$(dirname "$0")/arm_common.sh"
wt="$root/.claude/worktrees/codex-$label"
branch="codex-$label"

if [ -e "$wt" ]; then
  echo "codex_arm: $wt already exists; remove it or pick another label" >&2
  exit 3
fi

git -C "$root" worktree add "$wt" -b "$branch" "$base" >/dev/null
echo "codex_arm: worktree $wt at $(git -C "$root" rev-parse --short "$base")"

# Symlink installed deps rather than reinstalling: minutes saved per run, and the
# brief tells the worker not to install. Only link what the repo actually has.
for d in node_modules web/node_modules worker/node_modules; do
  [ -d "$root/$d" ] && ln -sfn "$root/$d" "$wt/$d"
done

# --disable plugins/plugin_sharing keeps an autonomous run on the narrow
# code-editing surface even when the local install has marketplace plugins.
# --dangerously-bypass-approvals-and-sandbox is the flag this version takes
# (NOT --yolo). 2-hour cap: a GUARD against a runaway, not a budget. 40 minutes
# was too short (Chris, 2026-08-19) — a real ticket that touches a store, its
# runner and an admin page runs past it, and a timeout produces exit 124 with a
# dirty tree and no commit, which is indistinguishable from the model failing.
scripts=$(cd "$(dirname "$0")" && pwd)

# Both arms get the same boundary. shadow_arm.sh gets it via pi_bench, which
# appends the same file; codex does not go through pi_bench, so it is stamped
# here. One file, so the two arms cannot drift apart.
brief_stamped=$(mktemp)
trap 'rm -f "$brief_stamped"' EXIT
cat "$brief" "$scripts/worker_rules.md" > "$brief_stamped"

# --json makes the run MEASURABLE: `turn.completed` carries the token breakdown,
# where the plain-text stream reports only a rounded total with no input/output
# split — useless at a 6x output multiplier. --output-last-message keeps the
# human-readable report, which the JSON stream would otherwise cost us.
logdir="$HOME/vault/data/model-bench/logs"
mkdir -p "$logdir"
jsonlog="$logdir/$(date +%Y%m%dT%H%M%S)-$label-codex.jsonl"
lastmsg="$logdir/$(date +%Y%m%dT%H%M%S)-$label-codex.final.md"

echo "codex_arm: running $model (2h cap), events -> $jsonlog"
start=$SECONDS
timeout 7200 codex exec --json --disable plugins --disable plugin_sharing \
  --dangerously-bypass-approvals-and-sandbox -C "$wt" \
  -o "$lastmsg" \
  -c model="$model" -c model_reasoning_effort="high" - < "$brief_stamped" > "$jsonlog"
status=$?
wall=$((SECONDS - start))

# A completion SENTINEL, so waiting on an arm never means grepping a process
# table. `pgrep -f '<pattern>'` matches the watcher's own command line whenever
# the pattern is written inline, so the arm reads as running forever and the
# watch stays silent past the moment it was armed for — that cost a whole
# turn on 2026-08-04, having already been diagnosed twice the same day.
# Wait on this file instead: it cannot match itself.
arm_done "$branch" "$status" "$wt"

# Record the CONTROL half of the pair from its own token counts, now, while the
# numbers exist. The ledger's first 27 rows have no control at all, so no cost
# question about them can be answered with a measurement — only with the
# challenger's volume repriced. Never block the arm on a bookkeeping failure.
python3 "$scripts/pi_bench.py" --record-control \
  --label "$label" --model "$model" \
  --from-codex-log "$jsonlog" --wall-s "$wall" --worktree "$wt" \
  || echo "codex_arm: WARNING control row not recorded (log kept at $jsonlog)" >&2

if [ -s "$lastmsg" ]; then
  echo "--- worker final message ---"
  cat "$lastmsg"
fi
echo "codex_arm: $label exited $status in ${wall}s (worktree $wt, branch $branch)"
exit $status

#!/usr/bin/env bash
# Gzip finished benchmark logs.
#
# pi_bench.py streams every model event to ~/vault/data/model-bench/logs/<stamp>-<label>.jsonl.
# A long arm writes gigabytes: on 2026-09-09 the directory held 35GB across 453 logs and the
# disk was full. The logs are evidence and stay, but a finished log is never appended to, so
# it can live gzipped. JSONL compresses about ten to one, and `gzip -dc` reads it back.
#
# A log is finished when nothing has written to it for --min-age-hours (default 24) and no
# process holds it open. The mtime check is the same "is it live" proxy the worktree sweep uses;
# the open-file check guards an arm that stalled rather than ended (see arm-safety.md).
#
# Usage:
#   compress_logs.sh                    # dry run, show what would go
#   compress_logs.sh --yes              # do it
#   compress_logs.sh --yes --min-age-hours 2
set -euo pipefail

dry=1
min_age_hours=24
logs="${BENCH_LOGS:-$HOME/vault/data/model-bench/logs}"

while [ $# -gt 0 ]; do
  case "$1" in
    --yes) dry=0; shift ;;
    --min-age-hours) min_age_hours="$2"; shift 2 ;;
    *) echo "unknown arg: $1" >&2; exit 2 ;;
  esac
done

[ -d "$logs" ] || { echo "no log directory at $logs" >&2; exit 1; }

if command -v pigz >/dev/null 2>&1; then zip=(pigz); else zip=(gzip); fi

compressed=0
skipped_open=0
before=0
after=0
while IFS= read -r -d '' f; do
  name="$(basename "$f")"
  if command -v fuser >/dev/null 2>&1 && fuser -s "$f" 2>/dev/null; then
    echo "skip (open)  $name"
    skipped_open=$((skipped_open + 1))
    continue
  fi
  size=$(stat -c %s "$f")
  before=$((before + size))
  if [ "$dry" -eq 1 ]; then
    echo "would compress  $name  ($((size / 1024 / 1024))MB)"
  else
    "${zip[@]}" "$f" && echo "compressed  $name"
    after=$((after + $(stat -c %s "$f.gz")))
  fi
  compressed=$((compressed + 1))
done < <(find "$logs" -maxdepth 1 -type f -name '*.jsonl' -mmin "+$((min_age_hours * 60))" -print0 | sort -z)

echo
echo "logs compressed: $compressed   skipped as still open: $skipped_open"
if [ "$dry" -eq 0 ]; then
  echo "size before: $((before / 1024 / 1024))MB   after: $((after / 1024 / 1024))MB"
else
  echo "size: $((before / 1024 / 1024))MB"
  echo "DRY RUN — re-run with --yes to act."
fi
exit 0

#!/usr/bin/env python3
"""Run a brief through `pi` in a worktree, measure it, append one ledger line.

  pi_bench.py <label> <worktree> <brief-file> [model]   # run + log (challenger)
  pi_bench.py --record-control --label L --model M \
      (--from-agent-transcript F | --from-codex-log F)
  pi_bench.py --compare                                 # cost/time/quality by control

Cost is computed here from token counts, NOT from a runner's own `cost` field —
pi falls back to default pricing for model ids missing from its catalog (the
dated `-0731` tags) and reports roughly 4x too high.

This file is the only place prices live. A copy anywhere else drifts silently,
because nothing compares the two — point at this table rather than restating it.
"""
import json, os, subprocess, sys, time
from datetime import datetime
from pathlib import Path

# USD per token: (uncached input, output, cached input).
#
# Challenger rates from https://openrouter.ai/api/v1/models; control rates are
# each vendor's published headline meter rate. Controls are priced here so a
# control arm's cost is MEASURED from its own token counts rather than
# reconstructed later from the challenger's volume — which is all the ledger
# could do for its first 27 rows, because nothing recorded the control at all.
PRICES = {
    # Challengers (OpenRouter)
    # The DATED tag, checked against OpenRouter's catalog 2026-08-04. It is
    # cheaper than the floating `-flash` alias (0.14/0.28) — check which one an
    # arm actually ran before pricing it.
    "deepseek/deepseek-v4-flash-0731": (0.09e-6, 0.18e-6, 0.018e-6),
    "deepseek/deepseek-v4-pro": (0.435e-6, 0.87e-6, 0.0087e-6),
    # Controls, headline meter rates (checked 2026-08-07). Cached input is 10%
    # of the uncached rate for every vendor here.
    "gpt-5.6-sol": (5e-6, 30e-6, 0.5e-6),
    "claude-fable-5": (10e-6, 50e-6, 1.0e-6),
    "claude-opus-5": (5e-6, 25e-6, 0.5e-6),
    "claude-opus-4-8": (5e-6, 25e-6, 0.5e-6),
    # Sonnet 5 headline is $3/$15; an intro rate of $2/$10 runs to 2026-08-31.
    # The headline is what a meter charges after that, so it is what we compare.
    "claude-sonnet-5": (3e-6, 15e-6, 0.3e-6),
}
# Outcomes a pair can be COMPARED on. `rejected` belongs here and its absence
# was the fourth silent bug in this ledger, all four erring the challenger's
# way (2026-08-21). Excluding it meant a pair whose challenger produced
# nothing usable vanished from the table entirely: the quality column could
# not report a failure even in principle, so it always read as some mix of
# clean and with-fixes, and the money spent on runs that delivered nothing
# was left out of the cost ratio. Seven pairs were hidden this way, five of
# them against gpt-5.6-sol controls. A comparison that cannot show a loss is
# not a comparison.
COMPARABLE = {"shipped-clean", "shipped-with-fixes", "rejected"}
DEFAULT_MODEL = "deepseek/deepseek-v4-flash-0731"
# Overridable so the write paths can be exercised without appending test rows to
# the real ledger. A recording script with no safe way to rehearse it gets
# rehearsed on production data, once, by someone in a hurry.
LEDGER = Path(os.environ.get("BENCH_LEDGER") or Path.home() / "vault/data/model-bench/runs.jsonl")
LOGS = LEDGER.parent / "logs"  # gitignored; a long run writes gigabytes, compress_logs.sh gzips finished ones


def cost_of(model, u):
    """USD for one run's usage, or None if the model has no price on file.

    Cache WRITES are charged as a multiple of the input rate rather than as their
    own column, because the multiple is a vendor fact rather than a per-model
    price: Anthropic bills 1.25x for a 5-minute entry and 2x for a 1-hour one,
    while OpenAI charges writes at the plain input rate. `u` may carry
    `cacheWrite5m` / `cacheWrite1h`; a run that reports neither is unaffected.

    This is not a rounding detail. One reconstructed control arm wrote 1.9M
    cache-creation tokens — about $12 at Opus rates — which a model that only
    counted reads would have silently dropped from the comparison.
    """
    pin, pout, pcache = PRICES.get(model, (None, None, None))
    if pin is None:
        return None
    w5, w1 = (1.25, 2.0) if model.startswith("claude-") else (1.0, 1.0)
    return (u["input"] * pin
            + u["output"] * pout
            + u["cacheRead"] * pcache
            + u.get("cacheWrite5m", 0) * pin * w5
            + u.get("cacheWrite1h", 0) * pin * w1)


def measure(log):
    """Sum usage over assistant messages, from the per-message `message_end`
    events rather than from `agent_end`'s list.

    `agent_end.messages` is only the messages still in the context WINDOW, so a
    run that compacted reports a fraction of what it spent — and the fraction
    lands in the ledger looking like an ordinary number, understating exactly
    the long runs that cost the most. Caught 2026-08-11: one challenger's
    agent_end held 2 assistant messages and 2,942 output tokens where
    message_end had 59 and 29,625, a ~10x understatement that would have been
    published as a 3,100x cost win over its control.

    The two agree exactly when nothing compacts, so this is strictly better
    rather than a trade. `agent_end` still supplies the run's final text and,
    by its absence, the `truncated` flag."""
    msgs, ends, turns, tools, final, stop = None, [], 0, 0, "", None
    for line in log.read_text(errors="replace").splitlines():
        if not line.startswith("{"):
            continue
        try:
            ev = json.loads(line)
        except json.JSONDecodeError:
            continue
        if ev.get("type") == "agent_end":
            msgs = ev.get("messages") or msgs
        elif ev.get("type") == "message_end":
            m = ev.get("message") or {}
            ends.append(m)
            # The LAST assistant stop reason. "length" means the model was cut off
            # mid-work by a context/output ceiling — a very different thing from
            # finishing badly, and invisible in `truncated`, which only detects a
            # killed process. Two runs were graded as capability failures on
            # 2026-08-03 when they had in fact hit this ceiling.
            if m.get("stopReason"):
                stop = m["stopReason"]
        elif ev.get("type") == "turn_end":
            turns += 1
            tools += len(ev.get("toolResults") or [])
    u = {k: 0 for k in ("input", "output", "cacheRead", "reasoning")}
    for m in ends if ends else (msgs or []):
        if m.get("role") != "assistant":
            continue
        for k in u:
            u[k] += (m.get("usage") or {}).get(k) or 0
    # The deliverable is whichever message ENDED the run, and agent_end is the
    # authority on that. Falling back to the message_end stream keeps a killed
    # run's last words.
    for m in msgs if msgs is not None else ends:
        if m.get("role") != "assistant":
            continue
        for c in m.get("content") or []:
            if c.get("type") == "text" and c.get("text"):
                final = c["text"]
    return u, turns, tools, final, msgs is None, stop


# Appended to EVERY arm's brief, control and challenger alike. Unlike COACHING
# below it is not levelling — it is the boundary of the job, so it applies to
# whoever is doing it. It lives in a plain file that this runner and the shell arm
# scripts both read, so the two arms cannot drift apart.
#
# A capable worker with spare time will helpfully do the coordinator's job — grade
# arms, compare branches, write the log — unless told the job has edges, and one
# that did so reported confidently on the wrong arm. Telling it is deterministic;
# remembering to tell it is not, which is why this lives in the runner.
WORKER_RULES = (Path(__file__).parent / "worker_rules.md").read_text()


# Appended to every challenger brief, automatically (Chris, 2026-08-01).
#
# It lives HERE rather than in the brief because the coordinator has to remember
# to paste it there, and demonstrably does not. Baked into the runner it cannot
# be forgotten, and every row in the ledger from COACHING_FROM onward had it.
#
# This deliberately makes the challenger's brief differ from the control's. That
# is levelling, not tilting: the Agent-tool control already gets this coaching
# free from its harness (skills, project CLAUDE.md, tool access) and the pi
# challenger has none of it. The question being measured is "how good is this
# model as a worker WITH a harness", not "which raw model wins on an identical
# prompt". Give each arm its best shot; grade the diff.
#
# Both instructions target one observed failure shape: the challenger does the
# literal task well and stops at its edge. See SKILL.md for the calibrating
# instances.
# BUMP THIS DATE whenever the COACHING text below changes. It is stamped onto
# every ledger row, so each generation of the instructions stays comparable
# against the others. Leaving it stale silently merges two different experiments.
COACHING_VERSION = "2026-08-21"
COACHING = """

---

## Standing instructions for this run

**Finish what you notice, or say plainly that you are not.** If you spot a
problem your change creates or exposes (a sentence that now reads badly, a value
that is now wrong, an adjacent case your fix makes reachable), fix it if it is
inside the spirit of this task, and flag it explicitly in your report if it is
not. What is not acceptable is noticing it, mentioning it in passing, and leaving
it half-done. Spotting a defect and reporting it is worth much less than spotting
it and closing it.

**Protect what survives.** Whatever you delete, refactor, or move, the code and
behaviour that REMAINS must end up at least as well covered as it started. If a
test file you remove was the only coverage of something that is staying, re-home
that coverage before you delete it. Report the test count before and after and
account for the difference explicitly: an unexplained drop will be read as lost
coverage, not as cleanup.

**Fix the class, not the instance.** Before you fix the thing the ticket names,
search the whole repo for every other place with the same defect, the same stale
string, the same pattern — every workspace, not only the one your file is in. A
worker string can be made untrue by a web change. List what you find, then fix
all of it. Half a sweep is not a smaller version of the right answer: the sites
you left behind now contradict the ones you changed.

#!/usr/bin/env bash
# Launch the challenger (shadow) arm for a delegated build, in one command.
#
# It exists because the control arm is a single Agent tool call while the shadow
# used to be three shell steps (worktree add, symlink node_modules, run the
# bench). That asymmetry is what made the shadow the thing that got skipped —
# twice on 2026-08-01 alone.
#
# Usage:
#   shadow_arm.sh <label> <brief-file> <base-ref> [model]
#
#   label      short slug, e.g. 212-dpa-status (also names the branch/worktree).
#              If that worktree already exists the label is auto-suffixed rather
#              than failing — see below.
#   brief-file the SAME brief handed to the control arm
#   base-ref   REQUIRED. The commit the control branched from. It has no default
#              on purpose: defaulting to HEAD means the start point is "whatever
#              main was when I happened to launch", which is not a comparison.
#   model      pi model tag (default: the bench script's default challenger)
#
# Run it from anywhere inside the repo. Backgrounding is the caller's job; use
# `setsid nohup ... &` so the run survives the harness tearing down its process
# group (which killed a whole overnight fleet on 2026-08-04).
set -euo pipefail

label=${1:?usage: shadow_arm.sh <label> <brief-file> <base-ref> [model]}
brief=${2:?usage: shadow_arm.sh <label> <brief-file> <base-ref> [model]}
base=${3:?usage: shadow_arm.sh <label> <brief-file> <base-ref> [model] — base-ref is required, so every arm starts from a commit you chose}
model=${4:-}

[ -f "$brief" ] || { echo "shadow_arm: no such brief: $brief" >&2; exit 2; }

root=$(git rev-parse --show-toplevel)
# shellcheck source=arm_common.sh
. "$(dirname "$0")/arm_common.sh"

# A name collision must not be a silent no-op: backgrounded, nobody reads the
# error, so two arms once "ran" for twenty minutes having died on launch. Suffix
# instead, and say which label was used.
orig_label=$label
suffix=1
while [ -e "$root/.claude/worktrees/ds4-$label" ]; do
  suffix=$((suffix + 1))
  label="${orig_label}-${suffix}"
  [ "$suffix" -le 20 ] || { echo "shadow_arm: too many worktrees named ds4-$orig_label*" >&2; exit 3; }
done
[ "$label" = "$orig_label" ] || echo "shadow_arm: ds4-$orig_label exists, using label $label instead"

wt="$root/.claude/worktrees/ds4-$label"
branch="ds4-$label"

# A worktree is fine BECAUSE of when we run: control and shadow go out together,
# so nothing of the control has landed on main and there is no answer to find.
# (Run a shadow after its control lands and the worktree's shared object database
# hands it `git show main:<path>` — that produced two byte-identical "results" on
# 2026-08-04. The fix is the timing, not git plumbing.)
git -C "$root" worktree add "$wt" -b "$branch" "$base" >/dev/null
base_sha=$(git -C "$root" rev-parse "$base")
echo "shadow_arm: worktree $wt at $(git -C "$root" rev-parse --short "$base")"

# Belt and braces alongside the isolation: say it in the brief too, since a reused
# brief may itself tell the worker to rebase onto something that is not there.
guarded=$(mktemp "${TMPDIR:-/tmp}/shadow-brief-XXXXXX.md")
trap 'rm -f "$guarded"' EXIT
{
  echo "# Your worktree's base commit is deliberate — do not move it"
  echo
  echo "You are on a branch based at \`$(git -C "$root" rev-parse --short "$base")\`, chosen on purpose."
  echo "Build against the tree as you find it: **do not rebase, merge, pull or reset**"
  echo "**onto \`main\`**, and do not read another branch for this ticket's answer. Any"
  echo "instruction below to rebase is stale and this overrides it."
  echo
  echo "---"
  echo
  cat "$brief"
} > "$guarded"

set +e
python3 "$(dirname "$0")/pi_bench.py" "$label" "$wt" "$guarded" ${model:+"$model"}
bench_status=$?
set -e

# See arm_common.sh: wait on this sentinel, never on a pgrep of the arm's own
# command line, which matches the watcher too and never goes quiet.
arm_done "$branch" "$bench_status" "$wt"

exit $bench_status

#!/usr/bin/env bash
# Sweep spent delegation worktrees.
#
# It exists because the "clean up after landing" rule in SKILL.md did not fire for
# the things that actually accumulated. On 2026-08-07 a repo held 33 worktrees; 24
# were SHADOW arms, which by design are never landed, so a rule phrased "after
# landing" never applied to them. A shadow's worktree is spent once it has been
# GRADED — the bench ledger is the record, not the working directory.
#
# Safety, in order:
#   - never touches a worktree it believes is live (see --min-age-hours)
#   - dry run unless --yes
#   - removes WORKTREES freely, but only deletes branches that are MERGED
#     (git branch -d refuses unmerged ones; unmerged shadow refs are kept, since
#     they hold the graded diff and cost nothing but a ref)
#
# Usage:
#   sweep_worktrees.sh                    # dry run, show what would go
#   sweep_worktrees.sh --yes              # do it
#   sweep_worktrees.sh --yes --min-age-hours 2
#   sweep_worktrees.sh --yes --keep foo --keep bar
set -euo pipefail

dry=1
min_age_hours=6
keep=()

while [ $# -gt 0 ]; do
  case "$1" in
    --yes) dry=0; shift ;;
    --min-age-hours) min_age_hours=${2:?}; shift 2 ;;
    --keep) keep+=("${2:?}"); shift 2 ;;
    -h|--help) sed -n '2,25p' "$0"; exit 0 ;;
    *) echo "sweep_worktrees: unknown arg: $1" >&2; exit 2 ;;
  esac
done

root=$(git rev-parse --show-toplevel)
cd "$root"
status_dir="$root/.claude/arm-status"
now=$(date +%s)
removed=0; kept_live=0

is_kept() {
  local n=$1
  for k in ${keep+"${keep[@]}"}; do [ "$k" = "$n" ] && return 0; done
  return 1
}

# Longest paths first so nested worktrees go before their parents.
while IFS=$'\t' read -r wt br; do
  [ "$wt" = "$root" ] && continue
  name=$(basename "$wt")
  short=${br#refs/heads/}

  if is_kept "$name" || is_kept "$short"; then
    echo "KEEP  $name (--keep)"; kept_live=$((kept_live + 1)); continue
  fi

  # An arm writes .claude/arm-status/<branch>.done when it exits. No sentinel plus a
  # recent commit means it is probably still running, so leave it alone. This is the
  # rule that makes the sweep safe to run while a build is in flight.
  if [ ! -f "$status_dir/$short.done" ]; then
    tip=$(git -C "$wt" log -1 --format=%ct 2>/dev/null || echo "$now")
    age_h=$(( (now - tip) / 3600 ))
    if [ "$age_h" -lt "$min_age_hours" ]; then
      echo "KEEP  $name (no sentinel, tip ${age_h}h old < ${min_age_hours}h — may be live)"
      kept_live=$((kept_live + 1)); continue
    fi
  fi

  dirty=$(git -C "$wt" status --porcelain 2>/dev/null | wc -l)
  note=""
  [ "$dirty" -gt 0 ] && note=" (discarding $dirty uncommitted files)"
  if [ "$dry" -eq 1 ]; then
    echo "would remove  $name$note"
  else
    git worktree remove --force "$wt" >/dev/null 2>&1 && echo "removed  $name$note"
  fi
  removed=$((removed + 1))
done < <(git worktree list --porcelain \
          | awk '/^worktree /{w=$2} /^branch /{print w"\t"$2}' \
          | awk '{print length, $0}' | sort -rn | cut -d' ' -f2-)

if [ "$dry" -eq 0 ]; then
  git worktree prune
  # Merged only: the content is already on main, so the ref is pure clutter.
  # git branch -d refuses anything unmerged, which is the guard, not an accident.
  git branch --merged main | grep -vE '^\*|^\s*main$' | while read -r b; do
    is_kept "$b" && continue
    git branch -d "$b" >/dev/null 2>&1 && echo "deleted branch  $b"
  done
fi

echo
echo "worktrees swept: $removed   kept as live: $kept_live"
echo "unmerged branches retained (graded shadow diffs): $(git branch --no-merged main | wc -l)"
[ "$dry" -eq 1 ] && echo "DRY RUN — re-run with --yes to act."
exit 0

#!/usr/bin/env bash
# Behaviour test for compress_logs.sh against a sandbox log directory.
set -euo pipefail
here="$(cd "$(dirname "$0")" && pwd)"
sandbox="$(mktemp -d)"
trap 'rm -rf "$sandbox"' EXIT
fail() { echo "FAIL: $1"; exit 1; }

echo '{"old":1}' > "$sandbox/20260801T000000-old.jsonl"
touch -d '3 days ago' "$sandbox/20260801T000000-old.jsonl"
echo '{"new":1}' > "$sandbox/20260909T000000-new.jsonl"
echo 'x' > "$sandbox/notes.txt"
touch -d '3 days ago' "$sandbox/notes.txt"

# Dry run changes nothing.
out="$(BENCH_LOGS="$sandbox" "$here/compress_logs.sh")"
[ -f "$sandbox/20260801T000000-old.jsonl" ] || fail "dry run compressed a file"
grep -q 'would compress  20260801T000000-old.jsonl' <<<"$out" || fail "dry run did not list the old log"
grep -q 'DRY RUN' <<<"$out" || fail "dry run did not say it was a dry run"

# --yes compresses only finished logs, keeps recent ones and non-log files.
BENCH_LOGS="$sandbox" "$here/compress_logs.sh" --yes >/dev/null
[ -f "$sandbox/20260801T000000-old.jsonl.gz" ] || fail "old log was not gzipped"
[ ! -f "$sandbox/20260801T000000-old.jsonl" ] || fail "old plain log still present"
[ "$(gzip -dc "$sandbox/20260801T000000-old.jsonl.gz")" = '{"old":1}' ] || fail "gzipped content differs"
[ -f "$sandbox/20260909T000000-new.jsonl" ] || fail "recent log was touched"
[ -f "$sandbox/notes.txt" ] || fail "non-log file was touched"

# A shorter age threshold catches the recent one too.
BENCH_LOGS="$sandbox" "$here/compress_logs.sh" --yes --min-age-hours 0 >/dev/null
[ -f "$sandbox/20260909T000000-new.jsonl.gz" ] || fail "min-age-hours 0 did not compress the recent log"

echo "PASS"

"""Tests for the pi_bench measurement layer.

The parser is the part worth pinning: a silent bug there produces plausible-
looking but wrong cost/token data, which is worse than no data at all.

Run: python3 -m pytest ~/.claude/skills/delegate/scripts/ -q
"""
import json
import os
import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent))
import pi_bench  # noqa: E402


def usage(inp=0, out=0, cache=0, reasoning=0):
    return {"input": inp, "output": out, "cacheRead": cache, "reasoning": reasoning}


def assistant(inp=0, out=0, cache=0, reasoning=0, text=None):
    content = [{"type": "text", "text": text}] if text else []
    return {"role": "assistant", "content": content,
            "usage": usage(inp, out, cache, reasoning)}


def write_log(tmp_path, events):
    p = tmp_path / "run.jsonl"
    p.write_text("".join(json.dumps(e) + "\n" for e in events))
    return p


def test_sums_usage_across_assistant_messages(tmp_path):
    log = write_log(tmp_path, [
        {"type": "agent_end", "messages": [
            assistant(inp=100, out=10, cache=5, reasoning=2),
            assistant(inp=200, out=20, cache=7, reasoning=3),
        ]},
    ])
    u, _, _, _, truncated, _stop = pi_bench.measure(log)
    assert u == {"input": 300, "output": 30, "cacheRead": 12, "reasoning": 5}
    assert truncated is False


def test_the_two_event_kinds_are_not_double_counted(tmp_path):
    """Both event kinds carry the same usage; counting both would double it."""
    msg = assistant(inp=100, out=10)
    log = write_log(tmp_path, [
        {"type": "message_end", "message": msg},
        {"type": "agent_end", "messages": [msg]},
    ])
    u, _, _, _, truncated, _stop = pi_bench.measure(log)
    assert u["input"] == 100 and u["output"] == 10
    assert truncated is False


def test_a_compacted_run_is_measured_whole(tmp_path):
    """agent_end carries only the messages still in the window.

    A run that compacted therefore reports a fraction of what it spent, and the
    fraction lands in the ledger looking like an ordinary number. Caught live on
    2026-08-11: a challenger's agent_end held 2 assistant messages and 2,942
    output tokens where message_end had 59 and 29,625, so the row understated
    the run by roughly ten times and made that arm look far cheaper than it was.
    """
    log = write_log(tmp_path, [
        {"type": "message_end", "message": assistant(inp=100, out=10)},
        {"type": "message_end", "message": assistant(inp=200, out=20)},
        {"type": "message_end", "message": assistant(inp=400, out=40)},
        # Compaction dropped the first two from the window.
        {"type": "agent_end", "messages": [assistant(inp=400, out=40)]},
    ])
    u, _, _, _, truncated, _stop = pi_bench.measure(log)
    assert u["input"] == 700 and u["output"] == 70
    assert truncated is False


def test_the_final_text_still_comes_from_agent_end(tmp_path):
    """Usage moved to message_end; the report text must not follow it there.

    agent_end's list is the authority on which message ENDED the run, and it is
    the run's deliverable, so it keeps coming from there.
    """
    log = write_log(tmp_path, [
        {"type": "message_end", "message": assistant(inp=1, text="early draft")},
        {"type": "agent_end", "messages": [assistant(inp=1, text="the report")]},
    ])
    _u, _, _, final, _truncated, _stop = pi_bench.measure(log)
    assert final == "the report"


def test_falls_back_to_message_end_when_killed_midrun(tmp_path):
    """A timeout kill leaves no agent_end — we still want the partial spend."""
    log = write_log(tmp_path, [
        {"type": "message_end", "message": assistant(inp=100, out=10)},
        {"type": "message_end", "message": assistant(inp=50, out=5)},
    ])
    u, _, _, _, truncated, _stop = pi_bench.measure(log)
    assert u["input"] == 150 and u["output"] == 15
    assert truncated is True


def test_ignores_user_messages(tmp_path):
    log = write_log(tmp_path, [
        {"type": "agent_end", "messages": [
            {"role": "user", "content": [], "usage": usage(inp=9999)},
            assistant(inp=100),
        ]},
    ])
    u, _, _, _, _, _stop = pi_bench.measure(log)
    assert u["input"] == 100


def test_counts_turns_and_tool_calls(tmp_path):
    log = write_log(tmp_path, [
        {"type": "turn_end", "toolResults": [{}, {}]},
        {"type": "turn_end", "toolResults": [{}]},
        {"type": "turn_end"},
        {"type": "agent_end", "messages": [assistant(inp=1)]},
    ])
    _, turns, tools, _, _, _stop = pi_bench.measure(log)
    assert turns == 3
    assert tools == 3


def test_returns_last_assistant_text(tmp_path):
    log = write_log(tmp_path, [
        {"type": "agent_end", "messages": [
            assistant(text="first"), assistant(text="final answer")]},
    ])
    _, _, _, final, _, _stop = pi_bench.measure(log)
    assert final == "final answer"


def test_skips_malformed_and_non_json_lines(tmp_path):
    p = tmp_path / "run.jsonl"
    p.write_text(
        'Warning: Model not found for provider "openrouter".\n'
        "{not valid json\n"
        + json.dumps({"type": "agent_end", "messages": [assistant(inp=42)]}) + "\n"
    )
    u, _, _, _, _, _stop = pi_bench.measure(p)
    assert u["input"] == 42


def test_missing_usage_fields_default_to_zero(tmp_path):
    log = write_log(tmp_path, [
        {"type": "agent_end", "messages": [{"role": "assistant", "content": []}]},
    ])
    u, _, _, _, _, _stop = pi_bench.measure(log)
    assert u == {"input": 0, "output": 0, "cacheRead": 0, "reasoning": 0}


def test_cost_uses_our_prices_not_pis(tmp_path):
    u = usage(inp=1_000_000, out=1_000_000, cache=1_000_000)
    cost = pi_bench.cost_of("deepseek/deepseek-v4-flash-0731", u)
    # The DATED tag's real prices (OpenRouter, verified 2026-08-04). Deliberately
    # NOT the floating alias's 0.14/0.28 — pinning those here is what made every
    # recorded cost ~55% too high until it was caught.
    assert cost == 0.09 + 0.18 + 0.018


def test_cost_is_none_for_unpriced_model():
    assert pi_bench.cost_of("some/unknown-model", usage(inp=100)) is None


# --- CLI flag parsing -------------------------------------------------------
# `_flag` was USED six times in the --record-control block and never defined, so
# every control-side recording died on `NameError: name '_flag' is not defined`.
# Nothing noticed because codex_arm.sh deliberately does not block an arm on a
# bookkeeping failure — it warns to stderr and exits 0. The result was a ledger
# of 47 rows without a single control row in it (found 2026-08-07).

def test_flag_reads_the_value_after_the_name():
    assert pi_bench._flag(["x", "--label", "229-race"], "--label") == "229-race"


def test_flag_is_none_when_absent():
    assert pi_bench._flag(["x", "--model", "m"], "--label") is None


def test_flag_casts_when_asked():
    assert pi_bench._flag(["x", "--wall-s", "616.5"], "--wall-s", float) == 616.5


def test_flag_is_none_when_name_is_last_and_has_no_value():
    # A truncated command line must not IndexError the whole run.
    assert pi_bench._flag(["x", "--label"], "--label") is None


# --- The review pass on STOP (Chris, 2026-08-19) ------------------------------
#
# The dominant failure in 57 coached flash runs was not a wrong diff, it was a
# run that stopped while the job was still half done: 10 swept part of a class,
# 3 finished cleanly and never committed at all. Telling the model "be thorough"
# in the brief has 57 rows of evidence against it, because the failure IS the
# stop. So the runner re-enters the worktree after the model believes it is
# finished and makes it look at its own work with a fresh context.

---

## The boundary of this job

**You are a builder. Build the ticket in front of you and stop there.**

- **Do not delegate.** Do not spawn sub-agents, launch other arms, or start
  another model to help. If the task is too large, say so in your report.
- **Do not write outside your own worktree.** Everything you create or modify
  belongs under the working directory you were given. In particular, never write
  to `~/vault/`, and never touch the project's delegation log (its `CLAUDE.md`
  names the path) or the bench ledger (`~/vault/data/model-bench/runs.jsonl`).
- **Do not grade, compare, or read other branches.** Another arm may be solving
  this same ticket in parallel. Reading it invalidates the comparison both runs
  exist to produce, and it is not your call to make. Judging arms and writing the
  experiment log are the coordinator's job.
- **Do not push.** Commit on your own branch and leave it there.
- **Use the dependency setup in your brief.** Keep dependency writes inside your worktree. Do not create writable dependency symlinks to another checkout. If dependencies are missing, follow the project setup instructions and report the setup cost.

## The gate you run: AFFECTED, never the whole suite

Every repo here defines exactly two gates, and its `CLAUDE.md` names the commands
for both. Read that before you run anything.

- **AFFECTED gate** — the tests your diff can actually reach, plus a typecheck for
  each area you touched. **This is the one you run.** In a vitest repo it is
  `--changed <BASE_REF>`, where `<BASE_REF>` is the base commit named in your
  brief; vitest walks the module graph from your changed files and skips every
  suite that cannot reach them.
- **FULL gate** — the whole suite, every workspace. **CI runs this on every push
  and is the backstop for everything you skip.** It is not your job.

**Never run a whole test suite.** Not `npm run check`, not `npm test --workspace
<name>`, not a bare `vitest run`. A suite-wide run proves nothing your affected
gate and CI do not already prove between them, and it is not free to anyone but
you: several arms share one machine, so the minutes you spend re-running the repo
are taken from every other arm on the box. On 2026-08-19 three arms each ran a
seven-minute suite at once on four cores; load average hit 22.8 and all three runs
were still unfinished fifteen minutes in.

**Careful — "scoped to a workspace" is not the affected gate.** A workspace's test
script usually carries no filter at all, so scoping that way is the full suite
under a smaller name. Only an explicit `--changed` (or an explicit file list)
narrows anything.

**One exception, and name it when you use it.** `--changed` follows imports, so it
under-selects when your diff changes something the module graph cannot see: a test
config, a shared harness or fixture, generated types. If you touched one of those,
run that area's whole suite once and say so in your report.

Your report is the deliverable: what you changed, the literal output of each gate
command, and the commit SHA.