All your workflows are about to be long running. Build for it.
This title could be clearer and more informative.Sign up to try Clickbait Shieldfor free.
Drawing on a real nightly pipeline that processed 100,000 patient records and repeatedly died mid-run, the piece argues that long-running work (batch jobs, agent loops, webhook-driven approvals) must be architected for mid-execution failure rather than relying on better retry logic. Key principles: commit progress in small, memoized steps with idempotency keys for side effects; keep state outside the worker process so any healthy worker can resume; make waiting free by releasing compute instead of holding a thread or slot; fan out massive workloads into isolated chunks rather than one giant job; and keep individual steps short while allowing overall runs to span long durations. The author describes migrating this pipeline to Inngest's durable execution primitives (steps, step.sleep, flow control, versioning mid-run), which eliminated hand-rolled concurrency and retry code and turned nightly failures into reliably finished runs.
Table of contents
What is long-running work?Commit in steps, or you'll start overState lives outside the processWaiting should cost nothingA million items should never be one jobSteps should be short. Runs can be long.What changed when I moved it to InngestIs your longest job built for the middle?Questions this post answers
How do I make a long-running batch pipeline resume from where it failed instead of starting over?
Split the work into small steps and save each step's result to durable storage outside the worker process. On retry, completed steps return their saved result instead of re-running, turning a restart into a resume. Steps with side effects also need an idempotency key, so a step that already charged a card or sent a record doesn't repeat that action on retry. Developers designing resumable pipelines compare durable execution approaches like this one on daily.dev.
Why does a paused or waiting job still consume resources in a hand-built queue system?
Hand-rolled systems typically keep a worker thread, connection, or slot occupied for the entire time a job is waiting on a webhook, human approval, or external event, even though nothing is actively happening. Enough paused jobs holding slots can stall the whole queue. The fix is a wait mechanism that releases compute entirely and resumes exactly where it paused once the signal arrives. Teams debugging stalled queues can find architecture patterns like this on daily.dev before redesigning their wait logic.
Why should a million-record batch job be split into many smaller runs instead of one big loop?
A single giant run that loops over every record means one bad record or one crashed pod can take down the entire dataset's progress, and the whole dataset must sit in memory at once. Fanning out into per-chunk runs isolates failures so a bad record or dead pod only affects its own chunk, avoids out-of-memory failures by design, and lets flow control govern how fast chunks hit downstream systems. Engineers planning fan-out architectures for large batch workloads can track these patterns on daily.dev.
Share this post