Agentic Editorial Pipeline

Part of the Paulopus case study. Paulopus needed a machine that could write. Every match wanted a pronostic before kickoff, a structured brief, and a debrief after the final whistle — researched against real sources, in a consistent voice, on a tournament's relentless schedule. The interesting engineering was never the prose model itself. It was the harness around it: what decides when to write, what is allowed to be written, and how the words get from a sandbox into the datastore without anyone getting to lie or overwrite the record. This is the story of that harness.

The runtime pivot: a routine, not an API harness

Decision. The original plan was a conventional generation harness — scripts/writer/generate.ts plus run.ts, calling claude-sonnet-4-6 with the web_search tool, driven by a GitHub Actions cron. Every run would bill against the Anthropic API. Before Phase 1 even shipped, I pivoted.

Approach. Instead of paying per token, I moved the whole generation loop onto a claude.ai scheduled routine running under a Max subscription. The routine does the research and the writing itself; the repo only exposes what work is due and how to write it back safely.

Artifact — from the decision log:

Result. The generation harness was never built. The routine follows a checked-in ROUTINE.md: run pnpm writer:due --api https://paulopus.vercel.app to pull due work and prepared prompts as JSON, adopt the persona, research each item (max 4 web searches per item, real https sources only, never fabricated), and write to a gitignored paulopus.ingest.json. API generation cost dropped to zero.

due.ts: gating due work by pure functions

Decision. A routine that wakes on a schedule must not try to write everything every time — it needs to compute exactly what has become due since it last ran, deterministically, so two runs never disagree about the state of the world.

Approach. due.ts derives due work through pure gates in lib/gates.ts, one per content kind: pronostics, match preps, detailed briefs, debriefs, and liminal/team prose. Each gate is a function of the match data and the clock, so the same inputs always produce the same verdict on what is owed.

Artifact — the gate families and the rationing rule, from the architecture summary:

gates:      pronostics · matchPreps · detailedBriefs · debriefs · liminals/teamProse
rationing:  locks > debriefs > earlyReads > detailedBriefs > liminals
budget:     MAX_ITEMS_PER_RUN = 4

Result. due.ts returns due items already rationed by priority and capped at four per run. Time-critical work — locking a pronostic before kickoff, debriefing a just-finished match — outranks slower-burning liminal prose, so under a backlog the routine always spends its budget where the clock is tightest.

ingest.ts: the one guarded write path

Decision. If the writer routine could scatter writes across the datastore, every integrity guarantee would depend on the model behaving. It cannot. There must be exactly one door in, and that door must enforce the rules regardless of what the routine hands it.

Approach. ingest.ts is the routine's only write path. It validates every incoming item against zod schemas and enforces integrity rules mechanically before anything is persisted: append-only-on-change, pre-kickoff-only, and https-only sources.

Artifact — the invariants ingest.ts enforces, from the architecture summary:

Result. The append-only-on-change semantics let pronostics become regenerable rather than write-once. An earlier design locked one immutable prediction; the shipped model keeps a drafts[] revision history revised up to kickoff on opening / brief-update / lineup-confirmed triggers — appended only on a real change. The schema wall makes that safe: history grows, but only with genuine, validated, pre-kickoff revisions.

The gzip/base64 dispatch bridge

Decision. During the live-tournament era the datastore was MongoDB Atlas, and the routine's cloud sandbox cannot open a raw TCP socket to it. The words had to reach Mongo without the routine ever touching Mongo.

Approach. I added a GitHub Actions workflow, ingest.yml, triggered by workflow_dispatch. The routine reads state over the public HTTPS API and writes by dispatching a payload — the ingest JSON, gzipped, then base64-encoded — to the workflow, which decodes it and runs the guarded pnpm writer:ingest against Mongo from inside CI, where full egress is available.

Artifact — the workflow contract, from the architecture summary:

.github/workflows/ingest.yml
  trigger: workflow_dispatch (payload: base64/gzip-encoded ingest JSON, required)
  purpose: decode the routine's dispatched payload and run guarded `pnpm writer:ingest`

Result. The gzip step was not there originally, and its absence caused a genuine silent failure: a full pronostics-plus-briefs run serialized to roughly 150KB as plain base64, over the workflow_dispatch input ceiling of about 65KB. The dispatch returned HTTP 422, created no run, and dropped the work with no error surfaced. The fix gzips before base64 and makes ROUTINE.md check the dispatch HTTP status and verify that a new run id actually appeared.

Rationing the queue down as scope narrowed

Decision. The pipeline started wider than it needed to be. One job had the routine automatically research and attach FIFA replay links to already-debriefed matches — but FIFA replays aren't embeddable or feed-discoverable, so that research was mostly guesswork.

Approach. I deleted the replay-backfill job entirely — its due gate, its prompt builder, and its ingest loop — and moved replay curation to a manual pnpm replay <matchId> <fifaUrl> script. In the same change I lowered the per-run budget.

Artifact — from the decision log:

MAX_ITEMS_PER_RUN came down from 6 to 4 at the same time.

Result. The queue got smaller and more honest: the routine only automates work it can do reliably from real sources, and a human curates the one thing the feed can't supply.

Retiring the pipeline into a static snapshot

Decision. Once the tournament ended, a live writer routine and a live database were pure liability — runtime secrets and moving parts with nothing left to write.

Approach. The final tournament data was snapshotted into data/matches.json (104 matches) and data/teams.json (48 teams); lib/db.ts was rewritten to read them in-memory behind unchanged async signatures. The Mongo-writing scripts were archived under scripts/_archive/, and the sync and ingest workflows were disabled with if: false.

Artifact — the workflows' end state, from the architecture summary:

Result. The app now runs off the static snapshot with no database and no runtime secrets, and the entire agentic pipeline sits archived and inert — its output frozen into 104 matches of prose, its machinery switched off cleanly rather than left running.