The Step That Survived the Crash

You did not build a retry. You built a system that remembers what it already paid for.

असाध्य तें साध्य करितां सायास · What Cannot Be Done Becomes Doable Through Practice

साधुनी बचनाग खाती तोळा तोळा । आणिकातें डोळां न पाहावे ॥ १ ॥
साधुनी भुजंग धरितील हातीं । आणिकें कापती देखोनियां ॥ २ ॥
असाध्य तें साध्य करितां सायास । कारण अभ्यास तुका म्हणे ॥ ३ ॥
Some have trained until they can swallow bachnag, the poison, by the tola, while others cannot bear to look at it. Some have trained until they hold a snake in the hand, while others shake at the sight of one. What cannot be done becomes doable through sustained effort, says Tuka, and the reason is practice.

The job that had already spent four hundred rupees

We had a nightly enrichment job. Fifteen records, one model call each, a database write after every one. It fell over on record fourteen because a vendor endpoint returned a 502, and the process exited the way processes do.

The next morning it ran again from record one.

Nobody had written a bug. Every line of that job was correct in isolation, and it did the only thing it knew how to do, which was start at the beginning. Thirteen model calls that had already succeeded, already been paid for, already produced good output, were made a second time. The inference bill was small enough that I noticed a week later rather than that morning, which is its own kind of finding. The failure was not expensive. It was invisible, and it had been happening for a while.

What I had built was a script that treated its own progress as worthless. Not deliberately. I had never named progress as a thing worth keeping, so nothing kept it.

The arithmetic nobody does before shipping

Here is the part that changed how I look at pipelines. Take a step that works ninety-nine times in a hundred. That is a good step. Chain five of them and your end-to-end success rate is about ninety-five percent. Chain ten and you are at ninety percent, which means one run in ten fails somewhere (Inngest).

Nothing in that math is exotic. It is multiplication, and it is the reason a workflow made entirely of reliable parts is not itself reliable. Reliability does not add up along a chain, it multiplies down.

Retries do not fix this, they relocate it. A retry at the top of a ten-step job re-runs the nine steps that worked to reach the one that did not. When those steps are pure computation the waste is cheap. When they are model calls, payment authorizations, or emails to a customer, the waste has a price and sometimes a witness. And the workloads people ship now run for hours, call external tools dozens of times, and pause on a human approval that lands the following morning. A process that must stay alive end to end is a bad fit for work shaped like that.

What the pattern actually does

Durable execution addresses this by moving the unit of durability from the job to the step.

The mechanism is unglamorous, which is the best thing about it. Every completed step appends its result to a persistent event history before the next one begins. If the worker dies, another worker picks the workflow up and replays that history: a step with a recorded result is not executed again, its value is handed back immediately, and real execution resumes at the first step with no entry. From inside your function it looks as though the code was suspended at an await and woken somewhere else, possibly on a different machine, possibly after a deploy (Temporal).

That single move converts my enrichment job from a script into something with a memory. Record fourteen fails, the workflow resumes, and records one through thirteen return from history without a second model call. The results were written down at the moment they existed, instead of being held in memory that a crash was always going to take.

This is also the shape of the transactional outbox, which solves a narrower version of the same problem. A service that writes to its database and then publishes an event is performing two writes with no atomicity between them, so a crash in the gap leaves a committed row nobody was told about. The outbox turns those two writes into one: the event is inserted into a table inside the same transaction as the business data, and a separate relay publishes it (AWS, Confluent). Same principle: put the record of the fact in durable storage in the same breath as the fact.

The tax, which is the honest part

Replay only works if replaying produces the same decisions, and that constraint reaches into how you write the code. Workflow logic must be deterministic: the same input must produce the same sequence of calls every time it runs (Temporal).

So the ordinary tools go on a list. A branch on the system clock is out, because replaying tomorrow takes the other path. A random number is out. Direct network calls, database queries, and model invocations leave the workflow body entirely and belong in activities whose results get recorded. Change the shape of a workflow while instances are mid-flight and replay can hit a command that does not match the history, which surfaces as a non-determinism error rather than a silent wrong answer.

This is a real constraint and I would rather state it plainly than sell around it. You give up ambient access to time, randomness, and I/O in one specific layer of your code. What you get is the ability to lose a machine without losing the work.

Where Tuka comes in

"साधुनी बचनाग खाती तोळा तोळा" is not a line about being tough. It is a line about what training does to a category. Bachnag is poison. It does not stop being poison because someone has trained on it. What changes is that a thing which would end an untrained person becomes something a practiced one can carry, by the tola, deliberately.

A crash is the poison here. It is not going away. No amount of uptime engineering removes the possibility that a machine disappears in the middle of step seven, and every architecture that assumes otherwise is quietly asking a process to stay alive as a correctness requirement. Durable execution does not prevent the crash. It makes the crash survivable, which is a different and much more available thing.

"आणिकें कापती देखोनियां" (others shake at the sight of it) reads to me now as a description of retry logic written in fear. The try blocks, the bookkeeping flags, the "did we already send this email" columns that accumulate in any long-lived job: that is what shaking looks like in code. Each one is a hand-rolled, partial durability layer, invented locally, tested rarely, and understood by one person who has since changed teams.

And "कारण अभ्यास" (the reason is practice) is the line I keep returning to, because the pattern's cost is front-loaded discipline. Determinism is a practice. Naming your steps is a practice. Refusing to read the clock inside workflow logic is a practice. None of it is clever, and the payoff arrives on the day something breaks, which is precisely the day you will not be in a position to invent it.

What I am reaching for

I am not going to put a workflow engine under a cron job that sorts a CSV. The pattern earns its cost where steps are expensive, externally visible, or slow, and the test is one question: if this dies halfway, what does it cost to start over, and who finds out? For the enrichment job the answer was thirteen paid calls and nobody. For a payment flow the answer is a duplicate charge and a customer.

What I changed is smaller than adopting a platform. Before writing any multi-step job now I ask which steps I would hate to repeat, and I make those record their own results before the next one starts. Sometimes that is a workflow engine. Often it is an outbox table and a status column with real names in it, done in an afternoon.

The job that ran fourteen records and then forgot all fourteen was not badly written. It was written by someone who had not yet decided that progress was worth storing. That decision is the whole pattern. The infrastructure is only the practice that follows it.

Sources

• Temporal, What is Durable Execution?.
• Temporal, Workflow Definition: determinism constraints and non-determinism errors.
• Inngest, Durable Execution: The Key to Harnessing AI Agents in Production, for the compound-reliability arithmetic.
• AWS Prescriptive Guidance, Transactional outbox pattern.
• Confluent, Understanding the Dual-Write Problem and Its Solutions.
• The abhanga is Tukaram, असाध्य तें साध्य करितां सायास, from the Tukaram Gatha.

Chetan Dhandal