Week 3 of 4 · Arc 5

The end-to-end harness

As of Oct 23, 2026 · W. Burch

By the end of week three a replay the agent can run replaces your click-through after each deploy. It covers Arc 5. The harness is a chain: a simulator of whatever your product consumes (telemetry, orders, documents, events), a capture that freezes one run into a fixture, a replay that feeds it to the deployed stack, assertions on what came out, and evidence kept for later. The cards use the layer table in Week one. The cards say what to build and why; the bootstrap prompt below has your agent build them, and you check its work against the cards.

The cards

Here, as of Oct 23: 41 regression fixtures plus 2 benchmark fixtures. 28 replay nightly in about 12 minutes and 31 in a full run. Start much smaller: one scenario, one fixture, one presence and one absence assertion, one workflow you start by hand.

Card 1

Every fault is a scenario that states its own answer

Enforce it, strongest first

  1. Test in CI: the simulator's own tests run each scenario twice with the same seed and require identical output.
  2. Code structure: each scenario is a small class that declares what moves, what stays flat, and what must and must not happen. The declaration is the source the assertions in card 4 are checked against.
text
scenario: payment_provider_times_out
seed: 7
moves:      payment provider latency 200 ms -> 30 s over phase 2
stays flat: order volume, inventory, the other providers
must happen:     order ord-0042 goes to payment_pending, one retry queued
must not happen: a second charge on ord-0042, any order marked failed

Card 2

A fixture is reproducible from its commit

Enforce it, strongest first

  1. CI check: fails a pull request that changes a scenario without recapturing its fixture.
  2. The capture script: refuses a dirty working tree, runs the scenario headless, and writes the records plus a summary of commit, seed and phase boundaries.
json
{"scenario": "payment_provider_times_out", "commit": "a1b2c3d", "seed": 7,
 "phases": [{"name": "baseline", "end_s": 300}, {"name": "fault", "end_s": 900}],
 "records": "payment_provider_times_out.jsonl"}

Card 3

Replay goes through the front door, into its own sandbox

Enforce it, strongest first

  1. Credentials: the replay signs in as an ordinary user, with no database, cluster or admin access, so it can only use the public API.
  2. Isolation: the replay script creates its own disposable sandbox (a tenant, account, namespace or preview environment), feeds the fixture in, pauses at each phase boundary to assert and to do any step a person would (approve, refund, sign off), and removes the sandbox at the end, pass or fail. Where the API can't create one, it uses a fixed test account and tags every record it makes, so cleanup can find and purge them.
text
replay payment_provider_times_out --env staging
  create sandbox replay-7f3a     (ordinary user credentials)
  feed fixture, phase baseline -> assert orders complete, no retries
  phase fault                  -> assert order pending, one retry; approve refund
  remove sandbox replay-7f3a     (on failure too)

Card 4

Assertions check outcomes, never prose

Enforce it, strongest first

  1. The replay's exit code: a per-fixture manifest with a fixed grammar (a record exists in a given state, a field has a given value, an event was or wasn't emitted). Every fixture asserts presence and absence: what must happen and what must not.
  2. Known failures: listed in the manifest with their ticket, and they flip to hard failures the day the bug stops reproducing, so a fix can't go unnoticed.
text
# payment_provider_times_out.expect
order ord-0042 state = payment_pending
retry queued order=ord-0042 count = 1
event payment.retry_scheduled emitted order=ord-0042
no second charge order=ord-0042
no order state = failed
known_failure PROJ-123: retry count reported as 2

Card 5

Evidence outlives the run

Enforce it, strongest first

  1. CI artifacts: each run uploads its manifest results, logs and timings, kept 30 days.
  2. An external scheduler starts the nightly run. Here the hosted CI cron ran hours late, so a scheduler outside CI dispatches the workflow instead.
text
nightly 02:00 UTC  ->  dispatch workflow e2e-replay (fixtures: regression)
keep artifacts 30 days: results.json, replay.log, timings.csv

Bootstrap prompt

This prompt carries out the five cards. Read the cards to know what good looks like, paste the prompt into your agent at the root of one repo, then check its gap report card by card. It builds the smallest version only and runs the replay once against a non-production environment, if you give it one. With no simulator and no such environment, it builds nothing for the replay and reports what each missing piece needs. It commits on a new local branch for you to review and never pushes.

text
Set up "week three" for running coding agents unattended in this repository:
the smallest end-to-end harness that replaces a person's click-through after
a deploy. Work through the steps in order. Nothing leaves this repo and
nothing is pushed.

1. Your harness. Name the agent tool you are running in and its version.
   Find its documentation (built-in help, local docs, or the official site;
   reading it online is fine). Quote only text you read word for word, label
   anything else as your own knowledge, and say "none" where it has none:
   a. how it runs a long command in the background and reads its result
      when it finishes, without polling;
   b. the longest a single foreground command may run;
   c. whether it can start a CI workflow and read that run's result.

2. This repo. Find what the product consumes (telemetry, events, orders,
   documents, requests), and where that input comes from in production.
   Find any simulator, generator, seed data or recorded fixtures; the
   public API the product exposes and how a client authenticates; every
   non-production environment and its URL; the CI config. Name the file
   you learned each fact from. List checks that already fail; don't fix
   them. Then decide what you can build, and say why:
   - No product code: build nothing.
   - Code but no simulator: build cards 1 and 2 only from the input
     schema you read in the code. Never invent an input schema.
   - No non-production environment with a public API: build nothing for
     cards 3 to 5.
   In every case still send step 3's questions; then the gap report says
   what each missing piece would need.

3. Questions. Put everything you could not find in ONE message to me:
   which non-production environment a replay may use and how to get
   ordinary-user credentials for it; which single fault to simulate
   first and what the product must do about it; whether a replay can
   create and remove its own sandbox (a tenant, account, namespace or
   preview environment) through the API, and if not, which fixed test
   account it may use instead. If nobody answers, continue on stated
   assumptions and mark each one ASSUMED in the final report. Never
   assume credentials or an environment URL; a missing one stops the
   replay, not the report.

4. The cards. Build the smallest version: one scenario, one fixture, one
   presence and one absence assertion, one workflow started by hand. For
   each card name the layer and the file, and propose the change as a diff.
   Card 1: a seeded, deterministic scenario that declares what moves,
           what stays flat, and what must and must not happen. Strongest
           here: a test that runs it twice with one seed and requires
           identical output.
   Card 2: a capture script that refuses a dirty tree and writes the
           records plus a summary of commit, seed and phase boundaries.
           Strongest here: a CI check that fails a pull request changing
           a scenario without recapturing its fixture. If the repo has no
           CI, put the check in the gap report instead of writing it.
   Card 3: a replay that uses only the public API, as an ordinary user,
           in its own sandbox: create it, feed in the fixture, assert and
           do any step a person would at each phase, and remove the
           sandbox at the end, pass or fail. With no sandbox to create,
           use the fixed test account, tag every record the replay makes,
           and purge the tagged records at the end.
   Card 4: assertions on outcomes, never on generated prose, in a
           per-fixture manifest with a fixed grammar; presence and
           absence both; known failures listed with a ticket, flipping to
           hard failures once they stop reproducing.
   Card 5: a CI workflow, started by hand, that runs the replay and keeps
           its results, logs and timings as artifacts for 30 days. Name
           what would schedule it nightly from outside CI; don't set it up.

5. Write the files on a new local branch, inside this repo only, and
   commit. If nobody answers, that commit is your proposal; I review it
   there. Never push or merge. Run the determinism test and the capture
   once (commit first, since capture refuses a dirty tree). Run the
   replay once against the non-production environment only if you have
   its URL and credentials; run it in the background, read its result,
   and show the output. Never run it against production. For anything
   outside the repo (credentials, CI secrets, the scheduler, the
   environment itself), write the exact steps for me instead.

6. End with a gap report, one row per card: card, layer used, file,
   whether it ran and what it printed, what is still missing, and every
   ASSUMED item.

Checklist

0 of 5 done

Ticks stay in this browser only.

Ready for week four when

  • After a deploy, the agent runs the replay and reports pass or fail; you don't click through.
  • A deliberately broken rule makes the replay fail on the absence assertion, not only the presence one.
  • A pull request that edits the scenario without recapturing is refused by CI.
  • The replay's sandbox, or every record it tagged, is gone after every run, including a failed one.