Journal-Loop A/B Ablation (2026-05-22, first run)

Placeholder. The CI workflow .github/workflows/journal-ablation.yml is dispatch-only and has not been triggered yet. Once a real run lands, the aggregator will commit a dated file (e.g. 2026-05-25-run-<id>.md) via a PR; this stub stays in tree as a beacon that the harness exists.

Methodology

Both arms run on the same git ref, same model, same prompts. The only difference is 0SEC_FEATURE_JOURNAL_LOOP (0 for native, 1 for journal).
Three slices: XBOW black-box, XBOW white-box, npm-bench. Each slice runs three repeats per arm so the table’s CIs are real (Wilson 95% for the found-rate proportion; normal-approx 95% for the continuous metrics).
Results aggregate raw per-run JSONs uploaded as workflow artifacts. The aggregator lives at .github/workflows/scripts/journal-ab-aggregate.mjs.

Promotion gate

The flag flips to default ON only when ALL of the following hold:

XBOW BB: journal found-rate journal_mean is >= 97/104 on the patched-104 lookback window. No statistical regression vs. native is acceptable here — this is the state-of-record slice.
Any slice: cost regression delta_pct on the cost ($) row is <= +10% (i.e. journal is at most 10% more expensive per run).
XBOW WB: wall-clock delta_pct on the wall (s) row is <= +50% (i.e. journal is at most 1.5x the native wall-clock).

If any of those checks fails, the flag stays OFF and the design doc records the gap.

Status note (audited 2026-08-22): the journalLoop flag and the single-point dispatcher described here were never built — there is no loop-dispatch.ts, and features.ts has no journalLoop entry. packages/core/src/agent/native-loop.ts remains the only agent loop; packages/core/src/agent/journal/ holds the journal primitives. This ablation protocol is retained as the plan of record, not as shipped wiring.

Results

slice	metric	native_mean	native_ci	journal_mean	journal_ci	delta_pct	winner
xbow-bb	found
xbow-bb	cost ($)
xbow-bb	wall (s)
xbow-wb	found
xbow-wb	cost ($)
xbow-wb	wall (s)
npm-bench	found
npm-bench	cost ($)
npm-bench	wall (s)

Decision

XBOW BB regression check
Cost regression check (<= 10% on all slices)
Wall-clock check on XBOW WB (<= 1.5x native)
Decision: flip default | keep OFF | rerun

Recorded by: Decision date: