There is a particular satisfaction that comes from understanding a system well enough to predict its failures. Most engineering writing skips the part where you didn't understand it yet — but that's usually the most useful part to read.

What follows is an attempt to write down the things I wish I'd known at the start, in the order that would have helped me. The code samples work; the assumptions are explicit; the conclusions are mine.

The setup

Every project of this kind starts with the same illusion: that the hard part is the algorithm, and once you have the algorithm, the rest is plumbing. The actual ratio, in my experience, runs about 15% algorithm and 85% plumbing, integration, and the strange edge cases that only show up when real users do real things.

1export async function process(input: Input): Promise<Result> {
2  const validated = await validate(input);
3  if (!validated.ok) return { kind: "error", reason: validated.reason };
4
5  const enriched = await enrich(validated.value);
6  const transformed = transform(enriched);
7
8  return persist(transformed);
9}

What changed

The right unit of retry is almost never the thing you started retrying. It's usually two layers down, or one layer up. Finding it is the work.

Once I started attaching cancellation tokens to every layer and propagating them through, an entire category of bug just stopped happening. The other bugs got easier to find. The system started feeling like a system.

Start with observability. Not "we'll add metrics later" observability — actual structured logs with trace IDs, before the first feature ships. The cost of adding it now is half a day. The cost of adding it after the first incident is a week, plus the incident.