Durable Agents and Sandboxing: Surviving the Crash

The Agent Refunded the Customer Twice

A refunds agent calls the payment API, and the pod is evicted before the next step runs. Your orchestrator does the sensible thing and restarts the job from the top. The model looks at the ticket, sees a refund is needed, and issues it again. Nobody wrote a bug, exactly. The system just treated a step with real consequences as if it could be retried for free.

Agents break the assumptions most of our infrastructure was built on. A run can last hours, wait two days for a manager's approval, and execute code that a model wrote a moment ago. A request handler with a try/catch can't deal with that. You need two properties: the run survives failure without repeating itself, and whatever it executes can't reach anything it shouldn't.

What Durable Execution Gives You

Durable execution means saving progress as you go, so a crashed run resumes from its last saved point rather than the beginning. In LangGraph that is a checkpointer. In Temporal it is a replayed event history. The mechanics differ and the contract is the same: a step whose result was saved is not run again. A crash between an external call and that save can still repeat the call, and that gap is why side effects need idempotency keys.

Check your durability setting, because the default may not be what you assume. LangGraph has three modes, and only one of them guarantees every step is saved:

I'd use sync for anything with side effects. A few milliseconds per step is nothing next to a double refund.

Checkpoints Don't Make a Side Effect Safe

This is the part that catches people. A checkpoint is saved after a step finishes, so a crash between the payment API returning and the checkpoint being written still leaves you with a refund nobody recorded. On resume, the step runs again. Only the system on the other end can settle that, which means you pass it an idempotency key.

from langgraph.checkpoint.postgres import PostgresSaver
from langgraph.func import entrypoint, task
from langgraph.types import Command, interrupt
from psycopg.rows import dict_row
from psycopg_pool import ConnectionPool

pool = ConnectionPool(DB_URL, kwargs={"autocommit": True, "row_factory": dict_row})
checkpointer = PostgresSaver(pool)
checkpointer.setup()  # first run only: creates the checkpoint tables

@task
def issue_refund(order_id: str, amount_cents: int, run_id: str) -> str:
    # The key comes from the run and the order, so a retry returns the first
    # refund instead of creating a second one. The task stops a replay from
    # repeating this; the key covers a crash the checkpoint never saw.
    # 'payments' stands in for your provider's client.
    refund = payments.refunds.create(
        order=order_id,
        amount=amount_cents,
        idempotency_key=f"refund:{run_id}:{order_id}",
    )
    return refund.id

@entrypoint(checkpointer=checkpointer)
def refund_workflow(request: dict) -> dict:
    if request["amount_cents"] > 50_000:
        approved = interrupt({"question": "Approve this refund?", "request": request})
        if not approved:
            return {"status": "rejected"}

    refund_id = issue_refund(request["order_id"], request["amount_cents"], request["run_id"]).result()
    return {"status": "refunded", "refund_id": refund_id}

# A stable thread_id is how a resume finds this run. durability="sync" saves
# each step before the next one starts.
config = {"configurable": {"thread_id": request["run_id"]}}
refund_workflow.invoke(request, config, durability="sync")

# Later, once an authenticated approver has decided. Check who is approving
# in your API, because Command(resume=...) won't do that for you.
refund_workflow.invoke(Command(resume=True), config, durability="sync")

One more trap sits in that interrupt. When a run resumes, the node restarts from its beginning, so any side effect placed before an interrupt() in the same node happens again. Put side effects in a @task or in a separate node after the approval. The rule I work by: a model call may repeat, but a side effect may not.

sequenceDiagram
    participant W as Workflow
    participant S as Checkpoint store
    participant P as Payment API
    W->>S: Save step 1
    W->>P: Refund, key refund:r42:o9
    P-->>W: Created
    Note over W: Pod evicted before the save
    W->>S: Load the last checkpoint
    W->>P: Refund, same key
    P-->>W: Returns the first refund
    W->>S: Save step 2
      

Choosing a Runtime

If the graph is the agent, LangGraph is enough. If the agent is one step in a business process with timers, compensation and long waits, I'd put it inside Temporal or a similar workflow engine, and treat each model call as an activity. The workflow engine then owns retries, and you avoid building a second, worse one around your agent. On Vercel, the Workflow SDK plays the same role.

The Sandbox Is the Real Boundary

Durability keeps a run alive. Sandboxing limits what a run can do, and for an agent that executes generated code it is the control that matters most. OWASP lists Unexpected Code Executionas ASI05, and treating the model as if it will always behave is how you end up there. A prompt injection in a fetched web page is enough to turn your helpful agent into someone else's.

Whatever you pick, network egress decides how bad a compromise gets. Deny by default, and allowlist the package registry and the APIs the task needs. Never put long-lived credentials inside the sandbox. If the agent has to call an authenticated API, route the call through a proxy on your side that adds the credential, so a compromised sandbox holds nothing worth stealing.

What Goes Wrong

What I Would Do First

List every tool your agent has and mark the ones that change something outside the agent. Each of those needs an idempotency key or a good reason it doesn't. Then kill your worker in the middle of a run in staging and watch what happens. Most teams learn more from that one test than from a week of design discussion.