By the end of week three a replay the agent can run replaces your click-through after each deploy. It covers Arc 5. The harness is a chain: a simulator of whatever your product consumes (telemetry, orders, documents, events), a capture that freezes one run into a fixture, a replay that feeds it to the deployed stack, assertions on what came out, and evidence kept for later. The cards use the layer table in Week one. The cards say what to build and why; the bootstrap prompt below has your agent build them, and you check its work against the cards.
The cards
Here, as of Oct 23: 41 regression fixtures plus 2 benchmark fixtures. 28 replay nightly in about 12 minutes and 31 in a full run. Start much smaller: one scenario, one fixture, one presence and one absence assertion, one workflow you start by hand.
Card 1
Every fault is a scenario that states its own answer
Test in CI: the simulator's own tests run each scenario twice with the same seed and require identical output.
Code structure: each scenario is a small class that declares what moves, what stays flat, and what must and must not happen. The declaration is the source the assertions in card 4 are checked against.
text
scenario: payment_provider_times_out
seed: 7
moves: payment provider latency 200 ms -> 30 s over phase 2
stays flat: order volume, inventory, the other providers
must happen: order ord-0042 goes to payment_pending, one retry queued
must not happen: a second charge on ord-0042, any order marked failed
CI check: fails a pull request that changes a scenario without recapturing its fixture.
The capture script: refuses a dirty working tree, runs the scenario headless, and writes the records plus a summary of commit, seed and phase boundaries.
Credentials: the replay signs in as an ordinary user, with no database, cluster or admin access, so it can only use the public API.
Isolation: the replay script creates its own disposable sandbox (a tenant, account, namespace or preview environment), feeds the fixture in, pauses at each phase boundary to assert and to do any step a person would (approve, refund, sign off), and removes the sandbox at the end, pass or fail. Where the API can't create one, it uses a fixed test account and tags every record it makes, so cleanup can find and purge them.
text
replay payment_provider_times_out --env staging
create sandbox replay-7f3a (ordinary user credentials)
feed fixture, phase baseline -> assert orders complete, no retries
phase fault -> assert order pending, one retry; approve refund
remove sandbox replay-7f3a (on failure too)
The replay's exit code: a per-fixture manifest with a fixed grammar (a record exists in a given state, a field has a given value, an event was or wasn't emitted). Every fixture asserts presence and absence: what must happen and what must not.
Known failures: listed in the manifest with their ticket, and they flip to hard failures the day the bug stops reproducing, so a fix can't go unnoticed.
text
# payment_provider_times_out.expect
order ord-0042 state = payment_pending
retry queued order=ord-0042 count = 1
event payment.retry_scheduled emitted order=ord-0042
no second charge order=ord-0042
no order state = failed
known_failure PROJ-123: retry count reported as 2
This prompt carries out the five cards. Read the cards to know what good looks like, paste the prompt into your agent at the root of one repo, then check its gap report card by card. It builds the smallest version only and runs the replay once against a non-production environment, if you give it one. With no simulator and no such environment, it builds nothing for the replay and reports what each missing piece needs. It commits on a new local branch for you to review and never pushes.
text
Set up "week three" for running coding agents unattended in this repository:
the smallest end-to-end harness that replaces a person's click-through after
a deploy. Work through the steps in order. Nothing leaves this repo and
nothing is pushed.
1. Your harness. Name the agent tool you are running in and its version.
Find its documentation (built-in help, local docs, or the official site;
reading it online is fine). Quote only text you read word for word, label
anything else as your own knowledge, and say "none" where it has none:
a. how it runs a long command in the background and reads its result
when it finishes, without polling;
b. the longest a single foreground command may run;
c. whether it can start a CI workflow and read that run's result.
2. This repo. Find what the product consumes (telemetry, events, orders,
documents, requests), and where that input comes from in production.
Find any simulator, generator, seed data or recorded fixtures; the
public API the product exposes and how a client authenticates; every
non-production environment and its URL; the CI config. Name the file
you learned each fact from. List checks that already fail; don't fix
them. Then decide what you can build, and say why:
- No product code: build nothing.
- Code but no simulator: build cards 1 and 2 only from the input
schema you read in the code. Never invent an input schema.
- No non-production environment with a public API: build nothing for
cards 3 to 5.
In every case still send step 3's questions; then the gap report says
what each missing piece would need.
3. Questions. Put everything you could not find in ONE message to me:
which non-production environment a replay may use and how to get
ordinary-user credentials for it; which single fault to simulate
first and what the product must do about it; whether a replay can
create and remove its own sandbox (a tenant, account, namespace or
preview environment) through the API, and if not, which fixed test
account it may use instead. If nobody answers, continue on stated
assumptions and mark each one ASSUMED in the final report. Never
assume credentials or an environment URL; a missing one stops the
replay, not the report.
4. The cards. Build the smallest version: one scenario, one fixture, one
presence and one absence assertion, one workflow started by hand. For
each card name the layer and the file, and propose the change as a diff.
Card 1: a seeded, deterministic scenario that declares what moves,
what stays flat, and what must and must not happen. Strongest
here: a test that runs it twice with one seed and requires
identical output.
Card 2: a capture script that refuses a dirty tree and writes the
records plus a summary of commit, seed and phase boundaries.
Strongest here: a CI check that fails a pull request changing
a scenario without recapturing its fixture. If the repo has no
CI, put the check in the gap report instead of writing it.
Card 3: a replay that uses only the public API, as an ordinary user,
in its own sandbox: create it, feed in the fixture, assert and
do any step a person would at each phase, and remove the
sandbox at the end, pass or fail. With no sandbox to create,
use the fixed test account, tag every record the replay makes,
and purge the tagged records at the end.
Card 4: assertions on outcomes, never on generated prose, in a
per-fixture manifest with a fixed grammar; presence and
absence both; known failures listed with a ticket, flipping to
hard failures once they stop reproducing.
Card 5: a CI workflow, started by hand, that runs the replay and keeps
its results, logs and timings as artifacts for 30 days. Name
what would schedule it nightly from outside CI; don't set it up.
5. Write the files on a new local branch, inside this repo only, and
commit. If nobody answers, that commit is your proposal; I review it
there. Never push or merge. Run the determinism test and the capture
once (commit first, since capture refuses a dirty tree). Run the
replay once against the non-production environment only if you have
its URL and credentials; run it in the background, read its result,
and show the output. Never run it against production. For anything
outside the repo (credentials, CI secrets, the scheduler, the
environment itself), write the exact steps for me instead.
6. End with a gap report, one row per card: card, layer used, file,
whether it ran and what it printed, what is still missing, and every
ASSUMED item.
Checklist
0 of 5 done
Ticks stay in this browser only.
Ready for week four when
After a deploy, the agent runs the replay and reports pass or fail; you don't click through.
A deliberately broken rule makes the replay fail on the absence assertion, not only the presence one.
A pull request that edits the scenario without recapturing is refused by CI.
The replay's sandbox, or every record it tagged, is gone after every run, including a failed one.