All chapters
  1. Start hereAn agent is a model call in a loop
  2. Chapter 1One call: what a model is from the outside
  3. Chapter 2The loop: where a call becomes an agent
  4. Chapter 3The abstraction ladder: six rungs over one call
  5. Chapter 4The eight jobs of a harness
  6. Chapter 5Context: deciding what the model sees
  7. Chapter 6Tools: what the model may ask for, and where it runs
  8. Chapter 7MCP: one way to plug tools into any harness
  9. Chapter 8RAG: retrieval is context assembly
  10. Chapter 9Control flow and state: stopping, pausing and resuming
  11. Chapter 10Guardrails: identity, approval and the output contract
  12. Chapter 11Reliability and cost: retries, budgets and the failures that will happen
  13. Chapter 12Observability and evals: knowing what the agent did, and whether it was right
  14. Chapter 13Many agents: more loops, and what each one costs
  15. Adopt · 1SDKs: the same loop in Python, TypeScript and Java
  16. Adopt · 2Coding harnesses: a finished harness for software work
  17. Adopt · 3Frameworks and graph runtimes: the loop as a library
  18. Adopt · 4Managed platforms: someone else runs the loop
  19. Adopt · 5Build your own: a whole harness in one file
  20. DecideBuild or adopt: who runs each job?
  21. ReferenceGlossary
  22. AboutAbout this site
Chapter 11

Reliability and cost: retries, budgets and the failures that will happen

Models time out, return bad output and cost money per step. Retries, fallbacks, idempotent side effects, budgets, and the failure modes to plan for.

The underwriter approved the reply. In step 8 the harness ran send_reply, and the quote for 18 Gulf Shore Blvd left for the broker. Then the process died. It had sent the email and had not yet saved the checkpoint that says so.

A minute later the run resumes. The newest checkpoint is the one from step 7. In it, the last message is the model's request to send the reply, with no result after it. The loop does what it is built to do with a pending tool request: it runs it.

Does the broker get two emails?

Nothing in this story is a bug. The crash was ordinary, and the resume did what Control and state designed it to do. A second email would come from two correct parts that were never told about each other.

Every recovery mechanism works by doing something again. A retry sends the request again, a resume runs the step again, a queue delivers the event again. So reliability is two questions for every failure: can doing it again succeed, and is doing it again safe? Each answer has a price.

This is job 7 of the eight: reliability and cost. The chapter breaks the miniature on purpose, one fault at a time, then collects the failures in one table, shows why cost belongs to the same job, and climbs to the products that take parts of it over.

1. A failed call is a status code, and the code says whether to try again#

From inside the loop, a failed model call is an HTTP response with a status that is not 200. The Claude API documents the ones that matter here.

StatusError type in the documentationCan the same request succeed later?
400invalid_request_errorNo. The request is wrong, and it will be wrong again
401authentication_errorNo. The key is wrong
429rate_limit_errorUsually, after a wait
500api_errorOften
504timeout_errorOften
529overloaded_errorOften. The page says a 529 "can occur when the API experiences high traffic across all users"

The first two rows are bugs or configuration errors, and sending the request again only hides them. The other four are conditions of the moment. So the first decision of job 7 is one line of code, and the miniature writes it exactly so: retryable = error.code == 429 or error.code >= 500.

2. A busy server: wait, then ask again#

The miniature's retry policy is two functions in examples/quote-email/harness.py, shown complete below. They demonstrate three decisions: which errors are retried, how long to wait, and when to change the model. A successful run proves that one failed request no longer ends the run. It does not prove the policy is right for a real outage; the waits were never tested against a live API.

def pick_model(failed_attempts: int) -> str:
    """Job 7: routing and fallback. After two failures, try the other model (assume its cache starts cold; the docs do not say)."""
    return PRIMARY_MODEL if failed_attempts < 2 else FALLBACK_MODEL


def call_with_retry(model, build_body) -> dict:
    """Job 7: retry 429 and 5xx with exponential backoff. Other errors are bugs; raise them."""
    for attempt in range(MAX_ATTEMPTS):
        try:
            return model.create(build_body(pick_model(attempt)))
        except urllib.error.HTTPError as error:
            retryable = error.code == 429 or error.code >= 500  # 529 = overloaded
            if not retryable or attempt == MAX_ATTEMPTS - 1:
                raise
            header = error.headers.get("retry-after") or ""  # seconds, or an HTTP date (then use backoff)
            wait = float(header) if header.replace(".", "", 1).isdigit() else BACKOFF_BASE_S * 2 ** attempt
            print(f"[retry] HTTP {error.code}, attempt {attempt + 1}, waiting {wait:.2f}s", file=sys.stderr)
            time.sleep(wait)
    raise AssertionError("unreachable")

Read the line that computes wait. If the server said how long to wait, the harness obeys. The rate limits page of the Claude API says: "When the response has a retry-after header, wait that many seconds before retrying." If there is no usable header, the wait doubles after each failure: 1 second, then 2, then 4, with BACKOFF_BASE_S at its live value of 1.0. That doubling is called exponential backoff. It exists because a server that is busy now is probably still busy a moment later, and a client that asks again at once adds to the load that caused the failure. MAX_ATTEMPTS is 4; after the fourth failure the function raises.

The command below runs the quote email against the scripted model with one fault: the first request is answered with a 429 that carries retry-after: 0.

cd examples/quote-email
/opt/homebrew/bin/python3 harness.py --fake --approve --verbose --fault 429
[retry] HTTP 429, attempt 1, waiting 0.00s
{"run_id": "run-721e31e0", "step": 1, "turn": 1, "kind": "model", "model": "claude-opus-5-5", "stop_reason": "tool_use", "input_tokens": 61, "output_tokens": 31, "cache_read": 0, "latency_ms": 0}
...
{"run_id": "run-721e31e0", "step": 9, "turn": 5, "kind": "model", "model": "claude-opus-5-5", "stop_reason": "end_turn", "input_tokens": 430, "output_tokens": 37, "cache_read": 526, "latency_ms": 0}

The ... stands for seven trace lines that match a run without the fault. The run ends with status done, the reply sent, 5 turns and 4,013 tokens: the totals of a clean run. The wait is 0.00 seconds because the scripted server asked for zero. The only sign of the failure is the first line, and it goes to the error stream, not to the trace file.

What this retry loop leaves outFour gaps a production policy must close

This is a simplified demonstration. Four things are missing, and each is found by reading the code, not by a run.

GapWhat happens in the miniatureWhat a production policy does
No jitterEvery client that failed at the same instant waits exactly 1, 2, 4 seconds and returns at the same instantAdd a small random extra wait, called jitter, so the retries spread out
No cap on the server's waitA large retry-after value is obeyed in fullCap the wait, and pause the run if the cap is not enough
Not every 429 is shortThe rate limits page says of the header: "Not every 429 carries it; the spend-cap 429 never does." The miniature retries every 429 the same wayTell a rate limit from a spending cap, and stop on the cap
Network errors are not caughtcall_with_retry catches urllib.error.HTTPError only. A dropped connection or a timeout on the 120-second limit in LiveModel.create raises a different exception and ends the processRetry connection errors and timeouts too. The earlier steps are still in the checkpoints

The full build had a bug of this family. Its first retry loop read retry-after as a number of seconds. The header may also carry a date. A date broke the loop, and nothing capped the wait. It went unnoticed because the header is rare. Failure-handling code runs rarely, so it needs tests of its own, which is why the scripted model can inject faults at all.

3. A model that stays overloaded: ask a different one#

Waiting helps when the failure is short. When the same model fails twice in a row, the miniature stops trusting it for this call: pick_model returns the second model from the third attempt on. A second model that the harness turns to when the first keeps failing is a fallback model.

The next command makes the scripted server answer 529 to every request for the primary model.

/opt/homebrew/bin/python3 harness.py --fake --approve --verbose --fault overloaded
[retry] HTTP 529, attempt 1, waiting 0.01s
[retry] HTTP 529, attempt 2, waiting 0.02s
{"run_id": "run-c62989ca", "step": 1, "turn": 1, "kind": "model", "model": "claude-sonnet-5-5", "stop_reason": "tool_use", "input_tokens": 61, "output_tokens": 31, "cache_read": 0, "latency_ms": 34}
{"run_id": "run-c62989ca", "step": 2, "turn": 1, "kind": "tool", "tool": "verify_address", "verdict": "allow", "is_error": false, "latency_ms": 0}
[retry] HTTP 529, attempt 1, waiting 0.01s
[retry] HTTP 529, attempt 2, waiting 0.02s
{"run_id": "run-c62989ca", "step": 3, "turn": 2, "kind": "model", "model": "claude-sonnet-5-5", "stop_reason": "tool_use", "input_tokens": 145, "output_tokens": 31, "cache_read": 526, "latency_ms": 43}
...

The run finishes with the same decision, and every trace line names claude-sonnet-5-5. The fake mode sets the base wait to 0.01; a live run would wait 1 and 2 seconds.

Now look at what repeats. The two failed attempts come back before each of the five model calls: ten requests to a model the harness had already seen fail. pick_model counts failures inside one call and forgets them when the call returns.

The missing piece is memory across calls. A circuit breaker is a counter that remembers recent failures of one target. When too many calls fail, it rejects new calls to that target at once, without an HTTP request, and lets a single test call through later. The full build has one; the miniature does not.

A fallback has three costs.

When every attempt fails, model_step sets the status to paused:model_unavailable and the loop saves a checkpoint. The run continues later with --resume RUN_ID, and only the failed call repeats. Pausing and resuming belong to Control and state. The point here is the order of escalation.

Diagram: A failed model call · four controls, by how long the failure lasts

Read left to right. Each column handles a failure that outlasted the column before it. The top box is the control; the box under it is what the control costs. Click any box.

  1. 1 · Seconds
    1. Wait, then send the same request: Retry with a wait that doubles after each failure, or the wait the server asked for.
    2. Costs waiting time: The run is slower. A retry that works leaves no mark unless it is recorded.
  2. 2 · Still failing
    1. Send it to a second model: After two failures in one call, the harness asks a different model.
    2. Costs different answers and a cold cache: The second model was not chosen for this step. Its first calls probably find no cached prefix.
  3. 3 · Minutes or hours
    1. Stop, and keep the finished work: Every attempt failed. The run is saved with a status that says why, and continues later by run id.
    2. Costs a run that waits for someone: Something has to notice the paused run and resume it. Only the failed call repeats.
  4. 4 · Never
    1. End in a named state: A failure that no retry can fix goes where a person can find it, with the reason.
    2. Costs a person's time: Someone inspects the case, fixes the cause and replays it.

Each column handles a longer failure and costs more. The last matters as much as the first: a failure that no control can fix must still end somewhere with a name, where a person can find it.

4. A reply that breaks the contract: a retry paid in tokens#

Not every failed call has an error status. The model can answer with status 200 and text the harness cannot use. In the miniature the final answer must be one JSON object with fixed fields. When it is not, the harness tells the model what was wrong and asks once more. The check and the repair message are job 6, and their home is Guardrails. The repair is here for one reason: it is a retry, and this kind always costs money.

/opt/homebrew/bin/python3 harness.py --fake --approve --verbose --fault bad_json
...
{"run_id": "run-2041d27a", "step": 9, "turn": 5, "kind": "model", "model": "claude-opus-5-5", "stop_reason": "end_turn", "input_tokens": 430, "output_tokens": 32, "cache_read": 526, "latency_ms": 0}
{"run_id": "run-2041d27a", "step": 10, "turn": 6, "kind": "model", "model": "claude-opus-5-5", "stop_reason": "end_turn", "input_tokens": 511, "output_tokens": 37, "cache_read": 526, "latency_ms": 0}

Steps 1 to 8 are elided. Step 9 is the answer that failed the check, and step 10 is the repaired one. The run ends done with 6 turns and 5,082 tokens, against 5 turns and 4,013 for a clean run. One repair made this run about a quarter larger, because the repair call resends the whole conversation, the bad answer included. The token numbers are the scripted model's estimates, not a provider's count.

A 429 retry repeats a request that produced nothing. A repair repeats a request that produced a full, billed response. That is why the miniature allows exactly one (MAX_REPAIRS = 1) and then ends with failed:invalid_output.

5. A run that spends too much: a budget checked between calls#

Retries and repairs add calls. A loop that does not converge adds many. Something has to say "enough". The miniature counts tokens: model_step adds the usage numbers of every response to state["tokens"], and the loop asks one function before every step. A limit on the tokens one run may use is a token budget.

def over_budget(state: dict) -> bool:
    """Job 7: token budget guard, fed by the usage numbers of every response."""
    return state["tokens"] >= state["token_budget"]

The default budget is 60,000 tokens. The command below sets it to 1,500, which the quote email cannot fit into.

/opt/homebrew/bin/python3 harness.py --fake --approve --token-budget 1500
{
  "run_id": "run-46d84706",
  "status": "stopped:token_budget",
  "decision": null,
  "reply": "none",
  "turn": 3,
  "tokens": 2115
}

The run stopped with a status that names the reason, and no email was sent. Now compare two numbers: the budget was 1,500 and the run used 2,115. The guard is not wrong. After the second call the total was 1,320, under the budget, so the third call was allowed, and that call alone used 795 tokens. The guard runs before a call and knows only what earlier calls cost.

So this kind of budget is a guard between steps, not a cap: it can be passed by the size of one call. It also counts tokens, not money; a token read from a cache counts the same as an output token here, though their prices differ. And a run that hits the budget has spent it and produced no decision. The control turns an unbounded loss into a bounded one. In the full build the caps are in dollars, per quote and per day, and a quote over a cap goes to a person.

6. The email that must not go twice#

Back to the opening. So far the harness repeated model calls, and repeating a model call changes nothing outside the run. Sending an email does. An action that changes the world outside the run is a side effect, and send_reply is the only one in the miniature.

First, the crash itself. --crash-after 8 ends the process in step 8, after the tool ran and before the checkpoint was written.

/opt/homebrew/bin/python3 harness.py --fake --approve --verbose --crash-after 8
...
{"run_id": "run-77f2aa5b", "step": 7, "turn": 4, "kind": "model", "model": "claude-opus-5-5", "stop_reason": "tool_use", "input_tokens": 328, "output_tokens": 51, "cache_read": 526, "latency_ms": 0}
{"run_id": "run-77f2aa5b", "step": 8, "turn": 4, "kind": "tool", "tool": "send_reply", "verdict": "allow", "is_error": false, "latency_ms": 0}
simulated crash in step 8, before its checkpoint. Continue with: --resume run-77f2aa5b

The trace line for step 8 says the send happened. The database agrees and disagrees: it holds checkpoints for steps 0 to 7 and nothing for step 8. To the loop, the email was never sent.

Now the resume. A successful run here proves one thing: that running step 8 a second time does not record a second email. It proves nothing about a real mail server, because the send in the miniature is simulated.

/opt/homebrew/bin/python3 harness.py --fake --approve --verbose --resume run-77f2aa5b
{"run_id": "run-77f2aa5b", "step": 8, "turn": 4, "kind": "tool", "tool": "send_reply", "verdict": "allow", "is_error": false, "latency_ms": 0}
{"run_id": "run-77f2aa5b", "step": 9, "turn": 5, "kind": "model", "model": "claude-opus-5-5", "stop_reason": "end_turn", "input_tokens": 434, "output_tokens": 37, "cache_read": 0, "latency_ms": 0}
{
  "run_id": "run-77f2aa5b",
  "status": "done",
  "decision": {
    "decision": "quote",
    "missing_fields": [],
    "reasons": [
      "eligible; flood zone AE"
    ],
    "premium_band": "high"
  },
  "reply": "sent",
  "turn": 5,
  "tokens": 4017
}

Step 8 ran again, with verdict allow. The harness did call send_once a second time. But the tool result saved in the step 8 checkpoint is {"status": "duplicate_suppressed", "idempotency_key": "91fe80a3cfa2d4ec"}, and the table of sent emails in runs.sqlite holds one row for this run. The broker gets one email. (The cache_read of 0 in step 9 is an artifact: the resumed process starts a new scripted model whose simulated cache is empty.)

Here is the function that made the second call harmless.

def send_once(db: sqlite3.Connection, run_id: str, email: dict) -> str:
    """Job 7: idempotency key. The same email in the same run is recorded once, so a resume cannot resend."""
    key = hashlib.sha256(json.dumps([run_id, email], sort_keys=True).encode()).hexdigest()[:16]
    inserted = db.execute("INSERT OR IGNORE INTO outbox VALUES (?, ?, ?)", (key, run_id, json.dumps(email)))
    db.commit()  # simulated send: a real system passes this key to a mail API that honours it, or uses an outbox worker
    return json.dumps({"status": "sent" if inserted.rowcount else "duplicate_suppressed", "idempotency_key": key})

It computes a short fingerprint from the run id and the full email: recipient, subject and body. It inserts a row under that fingerprint into a table whose primary key is the fingerprint. Then it reads the database's answer: one row inserted means this is the first time; zero rows means this exact email in this run was recorded before.

An operation that has the same effect whether it is performed once or five times is idempotent. A lookup is idempotent by nature. A send is not, so it has to be made so, and the usual way is the one above: give each intended effect a name that stays the same on every attempt, and let one store refuse to hold the name twice. That name is an idempotency key.

Diagram: The crash after the send · why the resume does not send a second email

Time runs down the page. The loop runs step 8 twice. The table of sent emails accepts the key once. Click any step.

  1. Harness loop to Checkpoints: Save step 7: send request pending. The model asked for send_reply. The request is saved before anything is sent, so a resume replays the same request with the same body.
  2. Harness loop to Outbox table: Step 8: insert the key. send_once hashes the run id and the email into a key and inserts a row under it.
  3. Outbox table to Harness loop: One row inserted: sent. The key was new, so this is the first send. In the miniature the insert is the send; it is simulated.
  4. Harness loop: Crash, before checkpoint 8. The process ends. The email is recorded as sent; the run state does not say so.
  5. Harness loop to Checkpoints: Resume: load the newest state. A new process loads the newest checkpoint for the run id and enters the same loop.
  6. Checkpoints to Harness loop: Step 7: send request still pending. To the loop the email was never sent. It runs the pending request again.
  7. Harness loop to Outbox table: Step 8 again: insert the same key. The same run id and the same stored email give the same key.
  8. Outbox table to Harness loop: No row inserted: duplicate suppressed. The primary key refuses the second row. send_once returns duplicate_suppressed, and nothing is sent.
  9. Harness loop to Checkpoints: Save step 8. The tool result is saved. The run continues to the final answer.

Notice who decides. The loop does not know whether this is a first attempt, and the model does not know a crash happened. The database decides, with a primary key, in one statement that cannot half-happen.

What goes into the key decides what "the same" means#

The key is a hash of [run_id, email]. Each part is a decision, and each can be wrong for a different system.

The payload is in the key, and a model wrote the payload. If the harness asked the model again after the crash, the model might word the body differently, the hash would differ, and a second email would go out. That does not happen here, for a reason outside send_once: the tool request was saved in the step 7 checkpoint, and the resume replays that stored request. The send is idempotent only because the request before it was checkpointed.

The run id is in the key, so the protection ends at the run. If the broker's email reaches the system twice and starts two runs, each run has its own id, and each sends a reply. The full build solves this one level up, with a key made from business facts: at intake, a key built from the address and the applicant, so that a request sent twice updates the same quote.

7. From a key in a table to an outbox#

The miniature's send is one database insert. That hides the hardest part of a real send.

A real harness has to do two things: record the key in its own database, and hand the email to a mail system. No single transaction covers both. A crash can fall between them, and the order decides what the crash costs.

OrderA crash between the two steps meansThe effect happens
Send first, then record the keyThe email went out, the key is missing, and the retry sends againAt least once
Record the key first, then sendThe key is there, the email never left, and the retry is suppressedAt most once

Neither is "exactly once". Two designs close the gap, and the comment in send_once names both.

Pass the key to the receiver. If the mail system's API accepts an idempotency key, the harness can retry freely and let the receiver drop the repeat. The duplicate is then stopped by the only party that knows for certain whether the email went out. This works only with a receiver that offers it.

Use an outbox. The harness never sends inside the step. In one transaction of its own database, it writes the new state of the run and a row that says "this email must be sent". Both commit or neither does, so a requested send can never be lost. A separate worker reads the rows, sends each email, and marks the row done. That table of effects waiting to be delivered is the outbox, and the design is called the outbox pattern.

The worker can crash too, after sending and before marking the row, and will then send again. So an outbox gives "at least once", and the key on each row turns that into "once" at a receiver that honours the key. The two ideas are used together.

By that definition, the miniature's table is an outbox in name and a key store in function. Its insert commits inside send_once, and the checkpoint commits later, in a separate transaction. That gap is where --crash-after 8 strikes. The simplification leaves out the worker and the shared transaction.

One rule survives every variation. Resume and redelivery make repeats certain, not rare. Every tool with a side effect needs a key, whichever product runs the loop.

8. In the full build: the same rules, with a queue between the stages#

The full build of the quote system is seven stages connected by six Kafka topics. The numbers in this section come from that build and were not re-measured here.

Kafka delivers an event at least once: if a stage crashes before it acknowledges an event, the event comes again. This is the opening scene with a different mechanism, and the answers are the same two. A dedupe table makes each stage's effects happen once, and each stage writes its outgoing event in the same transaction as its state change: the outbox pattern. A finished agent run is reused on redelivery, not paid for twice.

Some events fail every time. Retrying such an event in place would block every quote behind it. So each of the six topics has three retry topics with a growing delay, and then a dead-letter queue: a queue for events that could not be processed, where a person can inspect them and replay them once the cause is fixed. A malformed event skips the retries, since no delay will repair it.

The email that asks the broker for missing facts has a key too: a hash of the quote, the request version and the fields, claimed before the email is sent.

The full build's failure tableWhat fails, how the harness knows, what it does (from the source article)
What failsHow the harness knowsWhat it does
The model API is busy or downHTTP 429, 529 or 5xx, or a network errorThe SDK retries the same target. Then a gateway moves to the next target of the route. A circuit breaker skips a target that keeps failing
The request or configuration is wrongA client error such as HTTP 400 or 401Stop. No retry. An error record
The model refuses or runs out of roomThe stop reasonNo retry. To a person, with the reason
The model skips a required stepA post-condition in code, checked when the model says it is doneOne corrective turn, then a person
A tool rejects a callThe tool result is marked as an errorThe error text goes back to the model, so it can correct its call
A data service times outA timeout, 429, 5xx or I/O errorRetry with a growing wait: 200 ms × 2ⁿ plus jitter, or the server's Retry-After, capped at 10 seconds. A circuit breaker opens at 50% failures in 30 seconds and tests again after 60
A data service stays downRetries used up, or the breaker is openThe quote is marked partial. Only the failed service goes to a retry queue
The process crashes in a stageThe event was not acknowledgedRedelivery, with a dedupe table
An event keeps failingIt failed on every retryThree retry topics, then the dead-letter queue with a replay endpoint
A side effect could repeatA redelivered event or a resumed runA key, claimed before the email is sent
Two people act on one quoteThe version on the second screen is out of dateThe second write is rejected with HTTP 409

One service in this build is billed per call, so it gets zero retries.

9. Eight failures, side by side#

The miniature showed five failures. Four more need a paragraph each.

The model is slow, not down. No error arrives; the call takes minutes. The control is a time limit, and the miniature's live client sets one of 120 seconds. A call that is cut off has cost its input and returned nothing. For long outputs the Claude API errors page advises: "Consider using the streaming Messages API or Message Batches API for long-running requests, especially those over 10 minutes." Streaming adds a case to handle: "an error can occur after the API returns a 200 response".

A tool fails or times out. Tools showed the mechanics: run_tool_guarded returns the error as text, and the model can correct its request. The reliability question is who retries. A model that sees an error may ask again, at the price of a full model call; a failed lookup is cheaper to retry in code. A tool with a side effect must not be retried without a key, and a tool billed per call may deserve no retry at all.

The loop never converges. The model asks for the same tool with slightly different arguments, turn after turn. No single call fails. The control is a limit on turns, 8 in the miniature and 6 or 10 in the full build's two loops, with the budget as a second net. Control and state explains stop conditions.

A dependency returns stale data. No error, no delay. A cached property record is out of date, or two underwriters open one quote and the second saves over the first. Recovery makes this more likely, because a repeated step reads its inputs from caches and checkpoints. In the full build every quote carries a version, and a write from an out-of-date screen is rejected with HTTP 409.

Each row of the table starts with what a person watching the system would see.

The failureWhat you seeCauseControlCost of the control
The model is slow or unavailableCalls hang, or 5xx and 529Provider load or an incidentTime limit; retry with backoff; fallback model; pause the runWaiting; different answers and a cold cache on the fallback
Rate limited429Your own traffic passed a limitObey retry-after; back off; limit concurrencySlower runs; throughput has a ceiling
Malformed outputStatus 200, text that fails the checkThe model did not follow the contractOne repair turn, then a named failureA full extra model call
A tool fails or times outAn error result, or a long waitThe system behind the toolTime limit; retry reads in code; error text back to the modelExtra calls; an abandoned call may still be running
The run exceeds its budgetA growing bill, noticed lateRetries, repairs and long loops add callsToken or dollar budget between stepsOvershoot by one call; spent work with no result
A side effect is repeatedTwo emailsRetry, resume or redelivery after an effect, before its recordIdempotency key; outboxA table, a worker, a key design
The loop never convergesMany turns, no decisionNothing in the model's replies forces an endTurn limit; budget; route to a personThe cut-off run is wasted
A dependency returns stale dataA confident, wrong quoteCaches and replays serve old factsVersions and fetch times; reject stale writesRejected writes that someone must redo

Read the last column as one list. Every control costs time, money or work that someone must redo. Reliability does not remove failure. It chooses which cost to pay, in advance and in code.

10. Cost is a reliability problem#

Cost sits in job 7 because the same events drive both. Context showed the cause: the model keeps nothing between calls, so every step sends the whole context again. Add the usage fields of each model call in a clean run of the miniature, and the totals climb.

Model call12345Whole run
Tokens counted for the call6187027959059934,013

These are the scripted model's estimates, and the run is tiny. The shape carries over. The fifth call is about 60% larger than the first, because it resends four turns of history. An extra call late in a run is the most expensive kind. The repair turn in section 4 was one: 1,074 tokens, more than any call of the clean run.

Retries multiply the most expensive call. A repair, or a model that asks for a failed tool again, resends everything so far. The documentation read for this page does not say whether a request rejected with 429 or 529 is billed. A request that returned a response is.

A long loop costs more than its length. If every call is larger than the last, twice the turns cost more than twice as much.

Caching lowers the price and raises the ceiling. Prompt caching makes the unchanged prefix cheaper on every resend; Context has the arithmetic, including the price of writing to the cache. It also eases rate limits: the Claude API's rate limits page says the default input-token limit counts "only uncached input tokens".

For scale: the full build reports a model cost of about 0.4 cents for a quote with no flags and about 2.6 cents for a flagged one, cache writes and reads counted. The flagged quote enters a second agent loop.

11. The abstractions, each for the cause it removes#

Four functions did job 7 in the miniature. Each rung of the ladder takes some of them over. For each, ask the two questions of this chapter: does it try again for you, and does it make trying again safe?

A provider SDK retries the request#

The cause it removes: the hand-written loop of section 2 and its gaps.

The official Claude SDKs retry inside the call. The Python SDK page says: "By default, the SDK retries certain errors twice, with a short exponential backoff", and lists them: "connection errors (for example, a network problem), 408 Request Timeout, 409 Conflict, 429 Rate Limit, and >=500 Internal errors". The TypeScript and Java pages state the same default of 2, and all three give a default request timeout of 10 minutes. Errors arrive as typed exceptions: RateLimitError and InternalServerError in Python and TypeScript, RateLimitException and InternalServerException in Java.

The fragments below set both options. They are adapted from the official SDK documentation and were not run here.

Python

client = Anthropic(
    max_retries=4,   # default is 2; 0 turns retries off
    timeout=20.0,    # seconds; default is 10 minutes
)

TypeScript

const client = new Anthropic({
  maxRetries: 4,       // default is 2
  timeout: 20 * 1000,  // milliseconds; default is 10 minutes
});

Java

AnthropicClient client = AnthropicOkHttpClient.builder()
    .fromEnv()
    .maxRetries(4)                      // default is 2
    .timeout(Duration.ofSeconds(30))    // default is 10 minutes
    .build();

A timeout raises APITimeoutError in Python and APIConnectionTimeoutError in TypeScript, and both pages say timeouts are retried twice by default. The Java page does not say whether a timeout is retried.

What it costs: retries at two layers multiply. Put an SDK with 2 retries inside a loop like call_with_retry with 4 attempts, and one step can send 12 requests. Retry at one layer, and turn the other off.

What stays with you: the rest of this page. The provider SDK keeps no budget across calls and never runs a tool, so the email is as unprotected as before.

An agent SDK adds limits around the loop#

The cause it removes: nobody counts across calls. An agent SDK runs the loop, so it can count turns and money and change the model.

The Claude Agent SDK documents an option for each. fallback_model is the "Fallback model to use if the primary model fails". max_turns is "Maximum agentic turns". max_budget_usd will "Stop the query when the client-side cost estimate reaches this USD value". API retries are built in: CLAUDE_CODE_MAX_RETRIES is "maximum API retries. Default 10, capped at 15".

What it costs: the cost-tracking page says the cost fields "are client-side estimates, not authoritative billing data". It is the guard of section 5, in dollars. Idempotency for your own tools remains yours.

A framework turns each control into a component#

The cause it removes: reliability code tangled into the loop. An agent framework lets you add each control as a separate piece wrapped around the model call or the tool call, usually called middleware.

LangChain's built-in list reads like sections 2 to 5 of this page: ModelRetryMiddleware ("Automatically retry failed model calls with exponential backoff"), ToolRetryMiddleware, ModelFallbackMiddleware and ModelCallLimitMiddleware ("Limit the number of model calls to prevent infinite loops or excessive costs"). Others draw the line elsewhere. The OpenAI Agents SDK says "Retries are runtime-only and opt in". Microsoft Agent Framework's documentation shows retry and fallback as middleware you write.

What it costs: a component is easy to add without thinking. A tool-retry component around send_reply is the opening scene of this chapter, installed on purpose. Frameworks has a tab per product.

A model gateway offers the same controls from beside the ladder, as one service in front of every model call. LiteLLM's routing page lists "cooldowns, fallbacks, timeouts and retries (fixed + exponential backoff)". A cooldown is the memory pick_model lacked in section 3. The full build routes every model call through a gateway of its own.

A graph runtime or a durable execution engine retries whole steps#

The cause it removes: the process itself dying. These products save progress and run a failed step again; Control and state is their home. What matters here is the contract they state about repeats.

LangGraph has a RetryPolicy per node, with a default of 3 attempts. Its fault-tolerance page is plain about what a retry is: "A retry re-runs the entire node function from the start." That "gives at-least-once semantics for node execution", and "It does not make external side effects happen exactly once." The conclusion is in the vendor's words too: "Application code must make those effects idempotent."

Temporal, which calls itself "the open source Durable Execution platform for crash-proof applications and AI agents", says the same of its unit of work, the Activity: it "may be executed multiple times", and "Temporal recommends that Activities be idempotent". Its default retry policy for Activities starts at a 1-second interval, doubles, and has no limit on attempts: "The default is unlimited."

What it costs: an unlimited retry suits a database write and is dangerous for a model call that bills on every attempt. Set the attempt limit for anything that costs money, and give every side effect a key.

A managed platform runs the retries, and names the limits#

The cause it removes: operating any of this. The vendor runs the loop and handles the transient failures of its own model calls. How much the documentation says varies. Claude Managed Agents, in beta, has a session status rescheduling: "Transient error occurred, retrying automatically." It also has a real cap: "A session budget is an optional hard spend ceiling you set when you create a session", and a session at its budget "pauses and goes idle rather than terminating". Amazon Bedrock AgentCore's Harness has three limits per invocation: maxIterations (default 75), timeoutSeconds (default 3600) and maxTokens. OpenAI's Agents API says "OpenAI manages sessions, orchestration, context compaction, and recovery", and no more about retries.

What it costs: you cannot tune the policy, and the platform retries its own calls, not yours. Foundry Agent Service's limits page tells the caller to "Implement exponential backoff with jitter in your application retry logic". Managed platforms compares them.

12. Who runs job 7#

Two questions per product: who tries again, and who limits the spend. BUILT IN means the product does it in your process, MANAGED that the vendor runs it, YOU BUILD that it stays with you.

ProductRetry and fallbackLimits on spend
The miniatureYOU BUILD: call_with_retry, pick_modelYOU BUILD: over_budget, in tokens
Claude SDKs (Python, TypeScript, Java)BUILT IN: 2 retries with backoff, 10-minute timeout. Fallback: YOU BUILDYOU BUILD
Claude Agent SDKBUILT IN: API retries, fallback_modelBUILT IN: max_budget_usd (an estimate), max_turns
LangChain create_agentBUILT IN: model retry, tool retry and model fallback middlewareBUILT IN: limits on model calls and tool calls, not on money
LangGraphBUILT IN: RetryPolicy and a timeout per nodeThe pages read name no token or cost limit
Pydantic AIBUILT IN: ModelFallback, retriesBUILT IN: UsageLimits, for example a limit on requests
OpenAI Agents SDKBUILT IN: retries, opt in. No model fallback on the page readBUILT IN: max_turns
CrewAIBUILT IN: max_retry_limit, max_rpmBUILT IN: max_iter, max_execution_time
Microsoft Agent FrameworkYOU BUILD: the docs show retry and fallback as middleware you writeBUILT IN: max_function_calls, unlimited by default. A token or cost limit: the docs read do not say
Google ADKBUILT IN: a retry policy per node in graph workflowsBUILT IN: max_llm_calls, 500 by default
TemporalBUILT IN: automatic retries of ActivitiesNOT ITS JOB
LiteLLMBUILT IN: retries, fallbacks, cooldowns, timeoutsBUILT IN: spend tracking
Claude Managed AgentsMANAGED: transient errors retried automaticallyMANAGED: a session budget, a hard ceiling
Amazon Bedrock AgentCoreThe pages read do not describe retriesMANAGED: maxIterations, timeoutSeconds, maxTokens
OpenAI Agents APIMANAGED: "recovery" of sessions; no detail on the page read. Retrying a failed request: YOU BUILDThe page read describes no budget
Foundry Agent ServiceYOU BUILD: back off on 429 in your applicationThe docs read do not say

The Claude, LangGraph, LangChain, Temporal and managed-platform rows were read on 2026-11-16; the CrewAI row and the Microsoft Agent Framework retry cell come from the baseline check of 2026-11-14. The limits of Microsoft Agent Framework and Google ADK and the Pydantic AI fallback name are as on Frameworks. This is one column of the full matrix on Build or adopt.

The table has no column for "side effects happen once", and that is its most useful finding. No documentation read for this page fills it. Where it speaks, it hands the job back, as LangGraph and Temporal did. A product can repeat your step. Only you know what your step does to the world, and what its key is.

What you can do now, and what is still open#

You can now take any failure of an agent system and ask the two questions. Can doing it again succeed? Then retry, with a growing wait, a fallback and a limit. Is doing it again safe? For a model call and a lookup, yes. For an email, only with a key that names the effect and a store that refuses to hold the name twice. In the quote example the broker gets one email after the crash, and a run that spends too much stops with a status that says why.

A repeat is safe only when something can tell that it is a repeat. Every design in this chapter makes that one fact visible.

What is still open is how anyone knows. The retried 429 left one line on the error stream. The fallback changed the model for a whole run and changed no status. The repair made one run a quarter more expensive and still ended done. All of it worked, and none of it would be noticed. To see what a run did, and whether a change made the answers better or worse, the harness has to record and measure. That is job 8, and Observability and evals is next.