The underwriter approved the reply. In step 8 the harness ran send_reply, and the quote for 18 Gulf Shore Blvd left for the broker. Then the process died. It had sent the email and had not yet saved the checkpoint that says so.
A minute later the run resumes. The newest checkpoint is the one from step 7. In it, the last message is the model's request to send the reply, with no result after it. The loop does what it is built to do with a pending tool request: it runs it.
Does the broker get two emails?
Nothing in this story is a bug. The crash was ordinary, and the resume did what Control and state designed it to do. A second email would come from two correct parts that were never told about each other.
Every recovery mechanism works by doing something again. A retry sends the request again, a resume runs the step again, a queue delivers the event again. So reliability is two questions for every failure: can doing it again succeed, and is doing it again safe? Each answer has a price.
This is job 7 of the eight: reliability and cost. The chapter breaks the miniature on purpose, one fault at a time, then collects the failures in one table, shows why cost belongs to the same job, and climbs to the products that take parts of it over.
1. A failed call is a status code, and the code says whether to try again#
From inside the loop, a failed model call is an HTTP response with a status that is not 200. The Claude API documents the ones that matter here.
| Status | Error type in the documentation | Can the same request succeed later? |
|---|---|---|
| 400 | invalid_request_error | No. The request is wrong, and it will be wrong again |
| 401 | authentication_error | No. The key is wrong |
| 429 | rate_limit_error | Usually, after a wait |
| 500 | api_error | Often |
| 504 | timeout_error | Often |
| 529 | overloaded_error | Often. The page says a 529 "can occur when the API experiences high traffic across all users" |
The first two rows are bugs or configuration errors, and sending the request again only hides them. The other four are conditions of the moment. So the first decision of job 7 is one line of code, and the miniature writes it exactly so: retryable = error.code == 429 or error.code >= 500.
2. A busy server: wait, then ask again#
The miniature's retry policy is two functions in examples/quote-email/harness.py, shown complete below. They demonstrate three decisions: which errors are retried, how long to wait, and when to change the model. A successful run proves that one failed request no longer ends the run. It does not prove the policy is right for a real outage; the waits were never tested against a live API.
def pick_model(failed_attempts: int) -> str:
"""Job 7: routing and fallback. After two failures, try the other model (assume its cache starts cold; the docs do not say)."""
return PRIMARY_MODEL if failed_attempts < 2 else FALLBACK_MODEL
def call_with_retry(model, build_body) -> dict:
"""Job 7: retry 429 and 5xx with exponential backoff. Other errors are bugs; raise them."""
for attempt in range(MAX_ATTEMPTS):
try:
return model.create(build_body(pick_model(attempt)))
except urllib.error.HTTPError as error:
retryable = error.code == 429 or error.code >= 500 # 529 = overloaded
if not retryable or attempt == MAX_ATTEMPTS - 1:
raise
header = error.headers.get("retry-after") or "" # seconds, or an HTTP date (then use backoff)
wait = float(header) if header.replace(".", "", 1).isdigit() else BACKOFF_BASE_S * 2 ** attempt
print(f"[retry] HTTP {error.code}, attempt {attempt + 1}, waiting {wait:.2f}s", file=sys.stderr)
time.sleep(wait)
raise AssertionError("unreachable")
Read the line that computes wait. If the server said how long to wait, the harness obeys. The rate limits page of the Claude API says: "When the response has a retry-after header, wait that many seconds before retrying." If there is no usable header, the wait doubles after each failure: 1 second, then 2, then 4, with BACKOFF_BASE_S at its live value of 1.0. That doubling is called exponential backoff. It exists because a server that is busy now is probably still busy a moment later, and a client that asks again at once adds to the load that caused the failure. MAX_ATTEMPTS is 4; after the fourth failure the function raises.
The command below runs the quote email against the scripted model with one fault: the first request is answered with a 429 that carries retry-after: 0.
cd examples/quote-email
/opt/homebrew/bin/python3 harness.py --fake --approve --verbose --fault 429
[retry] HTTP 429, attempt 1, waiting 0.00s
{"run_id": "run-721e31e0", "step": 1, "turn": 1, "kind": "model", "model": "claude-opus-5-5", "stop_reason": "tool_use", "input_tokens": 61, "output_tokens": 31, "cache_read": 0, "latency_ms": 0}
...
{"run_id": "run-721e31e0", "step": 9, "turn": 5, "kind": "model", "model": "claude-opus-5-5", "stop_reason": "end_turn", "input_tokens": 430, "output_tokens": 37, "cache_read": 526, "latency_ms": 0}
The ... stands for seven trace lines that match a run without the fault. The run ends with status done, the reply sent, 5 turns and 4,013 tokens: the totals of a clean run. The wait is 0.00 seconds because the scripted server asked for zero. The only sign of the failure is the first line, and it goes to the error stream, not to the trace file.
What this retry loop leaves outFour gaps a production policy must close
This is a simplified demonstration. Four things are missing, and each is found by reading the code, not by a run.
| Gap | What happens in the miniature | What a production policy does |
|---|---|---|
| No jitter | Every client that failed at the same instant waits exactly 1, 2, 4 seconds and returns at the same instant | Add a small random extra wait, called jitter, so the retries spread out |
| No cap on the server's wait | A large retry-after value is obeyed in full | Cap the wait, and pause the run if the cap is not enough |
| Not every 429 is short | The rate limits page says of the header: "Not every 429 carries it; the spend-cap 429 never does." The miniature retries every 429 the same way | Tell a rate limit from a spending cap, and stop on the cap |
| Network errors are not caught | call_with_retry catches urllib.error.HTTPError only. A dropped connection or a timeout on the 120-second limit in LiveModel.create raises a different exception and ends the process | Retry connection errors and timeouts too. The earlier steps are still in the checkpoints |
The full build had a bug of this family. Its first retry loop read retry-after as a number of seconds. The header may also carry a date. A date broke the loop, and nothing capped the wait. It went unnoticed because the header is rare. Failure-handling code runs rarely, so it needs tests of its own, which is why the scripted model can inject faults at all.
3. A model that stays overloaded: ask a different one#
Waiting helps when the failure is short. When the same model fails twice in a row, the miniature stops trusting it for this call: pick_model returns the second model from the third attempt on. A second model that the harness turns to when the first keeps failing is a fallback model.
The next command makes the scripted server answer 529 to every request for the primary model.
/opt/homebrew/bin/python3 harness.py --fake --approve --verbose --fault overloaded
[retry] HTTP 529, attempt 1, waiting 0.01s
[retry] HTTP 529, attempt 2, waiting 0.02s
{"run_id": "run-c62989ca", "step": 1, "turn": 1, "kind": "model", "model": "claude-sonnet-5-5", "stop_reason": "tool_use", "input_tokens": 61, "output_tokens": 31, "cache_read": 0, "latency_ms": 34}
{"run_id": "run-c62989ca", "step": 2, "turn": 1, "kind": "tool", "tool": "verify_address", "verdict": "allow", "is_error": false, "latency_ms": 0}
[retry] HTTP 529, attempt 1, waiting 0.01s
[retry] HTTP 529, attempt 2, waiting 0.02s
{"run_id": "run-c62989ca", "step": 3, "turn": 2, "kind": "model", "model": "claude-sonnet-5-5", "stop_reason": "tool_use", "input_tokens": 145, "output_tokens": 31, "cache_read": 526, "latency_ms": 43}
...
The run finishes with the same decision, and every trace line names claude-sonnet-5-5. The fake mode sets the base wait to 0.01; a live run would wait 1 and 2 seconds.
Now look at what repeats. The two failed attempts come back before each of the five model calls: ten requests to a model the harness had already seen fail. pick_model counts failures inside one call and forgets them when the call returns.
The missing piece is memory across calls. A circuit breaker is a counter that remembers recent failures of one target. When too many calls fail, it rejects new calls to that target at once, without an HTTP request, and lets a single test call through later. The full build has one; the miniature does not.
A fallback has three costs.
- Different answers. The second model was not chosen for this step. Whether its quotes are as good is a question for measurement, which is job 8.
- A cold cache, probably. The documentation read for this page does not say that models share prompt-cache entries, so the code assumes they do not.
- Shared incidents. Two models at one provider can fail together. For its resolve step, the full build tries the same model at a second provider first, then a larger model.
When every attempt fails, model_step sets the status to paused:model_unavailable and the loop saves a checkpoint. The run continues later with --resume RUN_ID, and only the failed call repeats. Pausing and resuming belong to Control and state. The point here is the order of escalation.
Diagram: A failed model call · four controls, by how long the failure lasts
Read left to right. Each column handles a failure that outlasted the column before it. The top box is the control; the box under it is what the control costs. Click any box.
- 1 · Seconds
- Wait, then send the same request: Retry with a wait that doubles after each failure, or the wait the server asked for.
- Costs waiting time: The run is slower. A retry that works leaves no mark unless it is recorded.
- 2 · Still failing
- Send it to a second model: After two failures in one call, the harness asks a different model.
- Costs different answers and a cold cache: The second model was not chosen for this step. Its first calls probably find no cached prefix.
- 3 · Minutes or hours
- Stop, and keep the finished work: Every attempt failed. The run is saved with a status that says why, and continues later by run id.
- Costs a run that waits for someone: Something has to notice the paused run and resume it. Only the failed call repeats.
- 4 · Never
- End in a named state: A failure that no retry can fix goes where a person can find it, with the reason.
- Costs a person's time: Someone inspects the case, fixes the cause and replays it.
Each column handles a longer failure and costs more. The last matters as much as the first: a failure that no control can fix must still end somewhere with a name, where a person can find it.
4. A reply that breaks the contract: a retry paid in tokens#
Not every failed call has an error status. The model can answer with status 200 and text the harness cannot use. In the miniature the final answer must be one JSON object with fixed fields. When it is not, the harness tells the model what was wrong and asks once more. The check and the repair message are job 6, and their home is Guardrails. The repair is here for one reason: it is a retry, and this kind always costs money.
/opt/homebrew/bin/python3 harness.py --fake --approve --verbose --fault bad_json
...
{"run_id": "run-2041d27a", "step": 9, "turn": 5, "kind": "model", "model": "claude-opus-5-5", "stop_reason": "end_turn", "input_tokens": 430, "output_tokens": 32, "cache_read": 526, "latency_ms": 0}
{"run_id": "run-2041d27a", "step": 10, "turn": 6, "kind": "model", "model": "claude-opus-5-5", "stop_reason": "end_turn", "input_tokens": 511, "output_tokens": 37, "cache_read": 526, "latency_ms": 0}
Steps 1 to 8 are elided. Step 9 is the answer that failed the check, and step 10 is the repaired one. The run ends done with 6 turns and 5,082 tokens, against 5 turns and 4,013 for a clean run. One repair made this run about a quarter larger, because the repair call resends the whole conversation, the bad answer included. The token numbers are the scripted model's estimates, not a provider's count.
A 429 retry repeats a request that produced nothing. A repair repeats a request that produced a full, billed response. That is why the miniature allows exactly one (MAX_REPAIRS = 1) and then ends with failed:invalid_output.
5. A run that spends too much: a budget checked between calls#
Retries and repairs add calls. A loop that does not converge adds many. Something has to say "enough". The miniature counts tokens: model_step adds the usage numbers of every response to state["tokens"], and the loop asks one function before every step. A limit on the tokens one run may use is a token budget.
def over_budget(state: dict) -> bool:
"""Job 7: token budget guard, fed by the usage numbers of every response."""
return state["tokens"] >= state["token_budget"]
The default budget is 60,000 tokens. The command below sets it to 1,500, which the quote email cannot fit into.
/opt/homebrew/bin/python3 harness.py --fake --approve --token-budget 1500
{
"run_id": "run-46d84706",
"status": "stopped:token_budget",
"decision": null,
"reply": "none",
"turn": 3,
"tokens": 2115
}
The run stopped with a status that names the reason, and no email was sent. Now compare two numbers: the budget was 1,500 and the run used 2,115. The guard is not wrong. After the second call the total was 1,320, under the budget, so the third call was allowed, and that call alone used 795 tokens. The guard runs before a call and knows only what earlier calls cost.
So this kind of budget is a guard between steps, not a cap: it can be passed by the size of one call. It also counts tokens, not money; a token read from a cache counts the same as an output token here, though their prices differ. And a run that hits the budget has spent it and produced no decision. The control turns an unbounded loss into a bounded one. In the full build the caps are in dollars, per quote and per day, and a quote over a cap goes to a person.
6. The email that must not go twice#
Back to the opening. So far the harness repeated model calls, and repeating a model call changes nothing outside the run. Sending an email does. An action that changes the world outside the run is a side effect, and send_reply is the only one in the miniature.
First, the crash itself. --crash-after 8 ends the process in step 8, after the tool ran and before the checkpoint was written.
/opt/homebrew/bin/python3 harness.py --fake --approve --verbose --crash-after 8
...
{"run_id": "run-77f2aa5b", "step": 7, "turn": 4, "kind": "model", "model": "claude-opus-5-5", "stop_reason": "tool_use", "input_tokens": 328, "output_tokens": 51, "cache_read": 526, "latency_ms": 0}
{"run_id": "run-77f2aa5b", "step": 8, "turn": 4, "kind": "tool", "tool": "send_reply", "verdict": "allow", "is_error": false, "latency_ms": 0}
simulated crash in step 8, before its checkpoint. Continue with: --resume run-77f2aa5b
The trace line for step 8 says the send happened. The database agrees and disagrees: it holds checkpoints for steps 0 to 7 and nothing for step 8. To the loop, the email was never sent.
Now the resume. A successful run here proves one thing: that running step 8 a second time does not record a second email. It proves nothing about a real mail server, because the send in the miniature is simulated.
/opt/homebrew/bin/python3 harness.py --fake --approve --verbose --resume run-77f2aa5b
{"run_id": "run-77f2aa5b", "step": 8, "turn": 4, "kind": "tool", "tool": "send_reply", "verdict": "allow", "is_error": false, "latency_ms": 0}
{"run_id": "run-77f2aa5b", "step": 9, "turn": 5, "kind": "model", "model": "claude-opus-5-5", "stop_reason": "end_turn", "input_tokens": 434, "output_tokens": 37, "cache_read": 0, "latency_ms": 0}
{
"run_id": "run-77f2aa5b",
"status": "done",
"decision": {
"decision": "quote",
"missing_fields": [],
"reasons": [
"eligible; flood zone AE"
],
"premium_band": "high"
},
"reply": "sent",
"turn": 5,
"tokens": 4017
}
Step 8 ran again, with verdict allow. The harness did call send_once a second time. But the tool result saved in the step 8 checkpoint is {"status": "duplicate_suppressed", "idempotency_key": "91fe80a3cfa2d4ec"}, and the table of sent emails in runs.sqlite holds one row for this run. The broker gets one email. (The cache_read of 0 in step 9 is an artifact: the resumed process starts a new scripted model whose simulated cache is empty.)
Here is the function that made the second call harmless.
def send_once(db: sqlite3.Connection, run_id: str, email: dict) -> str:
"""Job 7: idempotency key. The same email in the same run is recorded once, so a resume cannot resend."""
key = hashlib.sha256(json.dumps([run_id, email], sort_keys=True).encode()).hexdigest()[:16]
inserted = db.execute("INSERT OR IGNORE INTO outbox VALUES (?, ?, ?)", (key, run_id, json.dumps(email)))
db.commit() # simulated send: a real system passes this key to a mail API that honours it, or uses an outbox worker
return json.dumps({"status": "sent" if inserted.rowcount else "duplicate_suppressed", "idempotency_key": key})
It computes a short fingerprint from the run id and the full email: recipient, subject and body. It inserts a row under that fingerprint into a table whose primary key is the fingerprint. Then it reads the database's answer: one row inserted means this is the first time; zero rows means this exact email in this run was recorded before.
An operation that has the same effect whether it is performed once or five times is idempotent. A lookup is idempotent by nature. A send is not, so it has to be made so, and the usual way is the one above: give each intended effect a name that stays the same on every attempt, and let one store refuse to hold the name twice. That name is an idempotency key.
Diagram: The crash after the send · why the resume does not send a second email
Time runs down the page. The loop runs step 8 twice. The table of sent emails accepts the key once. Click any step.
- Harness loop to Checkpoints: Save step 7: send request pending. The model asked for send_reply. The request is saved before anything is sent, so a resume replays the same request with the same body.
- Harness loop to Outbox table: Step 8: insert the key. send_once hashes the run id and the email into a key and inserts a row under it.
- Outbox table to Harness loop: One row inserted: sent. The key was new, so this is the first send. In the miniature the insert is the send; it is simulated.
- Harness loop: Crash, before checkpoint 8. The process ends. The email is recorded as sent; the run state does not say so.
- Harness loop to Checkpoints: Resume: load the newest state. A new process loads the newest checkpoint for the run id and enters the same loop.
- Checkpoints to Harness loop: Step 7: send request still pending. To the loop the email was never sent. It runs the pending request again.
- Harness loop to Outbox table: Step 8 again: insert the same key. The same run id and the same stored email give the same key.
- Outbox table to Harness loop: No row inserted: duplicate suppressed. The primary key refuses the second row. send_once returns duplicate_suppressed, and nothing is sent.
- Harness loop to Checkpoints: Save step 8. The tool result is saved. The run continues to the final answer.
Notice who decides. The loop does not know whether this is a first attempt, and the model does not know a crash happened. The database decides, with a primary key, in one statement that cannot half-happen.
What goes into the key decides what "the same" means#
The key is a hash of [run_id, email]. Each part is a decision, and each can be wrong for a different system.
The payload is in the key, and a model wrote the payload. If the harness asked the model again after the crash, the model might word the body differently, the hash would differ, and a second email would go out. That does not happen here, for a reason outside send_once: the tool request was saved in the step 7 checkpoint, and the resume replays that stored request. The send is idempotent only because the request before it was checkpointed.
The run id is in the key, so the protection ends at the run. If the broker's email reaches the system twice and starts two runs, each run has its own id, and each sends a reply. The full build solves this one level up, with a key made from business facts: at intake, a key built from the address and the applicant, so that a request sent twice updates the same quote.
7. From a key in a table to an outbox#
The miniature's send is one database insert. That hides the hardest part of a real send.
A real harness has to do two things: record the key in its own database, and hand the email to a mail system. No single transaction covers both. A crash can fall between them, and the order decides what the crash costs.
| Order | A crash between the two steps means | The effect happens |
|---|---|---|
| Send first, then record the key | The email went out, the key is missing, and the retry sends again | At least once |
| Record the key first, then send | The key is there, the email never left, and the retry is suppressed | At most once |
Neither is "exactly once". Two designs close the gap, and the comment in send_once names both.
Pass the key to the receiver. If the mail system's API accepts an idempotency key, the harness can retry freely and let the receiver drop the repeat. The duplicate is then stopped by the only party that knows for certain whether the email went out. This works only with a receiver that offers it.
Use an outbox. The harness never sends inside the step. In one transaction of its own database, it writes the new state of the run and a row that says "this email must be sent". Both commit or neither does, so a requested send can never be lost. A separate worker reads the rows, sends each email, and marks the row done. That table of effects waiting to be delivered is the outbox, and the design is called the outbox pattern.
The worker can crash too, after sending and before marking the row, and will then send again. So an outbox gives "at least once", and the key on each row turns that into "once" at a receiver that honours the key. The two ideas are used together.
By that definition, the miniature's table is an outbox in name and a key store in function. Its insert commits inside send_once, and the checkpoint commits later, in a separate transaction. That gap is where --crash-after 8 strikes. The simplification leaves out the worker and the shared transaction.
One rule survives every variation. Resume and redelivery make repeats certain, not rare. Every tool with a side effect needs a key, whichever product runs the loop.
8. In the full build: the same rules, with a queue between the stages#
The full build of the quote system is seven stages connected by six Kafka topics. The numbers in this section come from that build and were not re-measured here.
Kafka delivers an event at least once: if a stage crashes before it acknowledges an event, the event comes again. This is the opening scene with a different mechanism, and the answers are the same two. A dedupe table makes each stage's effects happen once, and each stage writes its outgoing event in the same transaction as its state change: the outbox pattern. A finished agent run is reused on redelivery, not paid for twice.
Some events fail every time. Retrying such an event in place would block every quote behind it. So each of the six topics has three retry topics with a growing delay, and then a dead-letter queue: a queue for events that could not be processed, where a person can inspect them and replay them once the cause is fixed. A malformed event skips the retries, since no delay will repair it.
The email that asks the broker for missing facts has a key too: a hash of the quote, the request version and the fields, claimed before the email is sent.
The full build's failure tableWhat fails, how the harness knows, what it does (from the source article)
| What fails | How the harness knows | What it does |
|---|---|---|
| The model API is busy or down | HTTP 429, 529 or 5xx, or a network error | The SDK retries the same target. Then a gateway moves to the next target of the route. A circuit breaker skips a target that keeps failing |
| The request or configuration is wrong | A client error such as HTTP 400 or 401 | Stop. No retry. An error record |
| The model refuses or runs out of room | The stop reason | No retry. To a person, with the reason |
| The model skips a required step | A post-condition in code, checked when the model says it is done | One corrective turn, then a person |
| A tool rejects a call | The tool result is marked as an error | The error text goes back to the model, so it can correct its call |
| A data service times out | A timeout, 429, 5xx or I/O error | Retry with a growing wait: 200 ms × 2ⁿ plus jitter, or the server's Retry-After, capped at 10 seconds. A circuit breaker opens at 50% failures in 30 seconds and tests again after 60 |
| A data service stays down | Retries used up, or the breaker is open | The quote is marked partial. Only the failed service goes to a retry queue |
| The process crashes in a stage | The event was not acknowledged | Redelivery, with a dedupe table |
| An event keeps failing | It failed on every retry | Three retry topics, then the dead-letter queue with a replay endpoint |
| A side effect could repeat | A redelivered event or a resumed run | A key, claimed before the email is sent |
| Two people act on one quote | The version on the second screen is out of date | The second write is rejected with HTTP 409 |
One service in this build is billed per call, so it gets zero retries.
9. Eight failures, side by side#
The miniature showed five failures. Four more need a paragraph each.
The model is slow, not down. No error arrives; the call takes minutes. The control is a time limit, and the miniature's live client sets one of 120 seconds. A call that is cut off has cost its input and returned nothing. For long outputs the Claude API errors page advises: "Consider using the streaming Messages API or Message Batches API for long-running requests, especially those over 10 minutes." Streaming adds a case to handle: "an error can occur after the API returns a 200 response".
A tool fails or times out. Tools showed the mechanics: run_tool_guarded returns the error as text, and the model can correct its request. The reliability question is who retries. A model that sees an error may ask again, at the price of a full model call; a failed lookup is cheaper to retry in code. A tool with a side effect must not be retried without a key, and a tool billed per call may deserve no retry at all.
The loop never converges. The model asks for the same tool with slightly different arguments, turn after turn. No single call fails. The control is a limit on turns, 8 in the miniature and 6 or 10 in the full build's two loops, with the budget as a second net. Control and state explains stop conditions.
A dependency returns stale data. No error, no delay. A cached property record is out of date, or two underwriters open one quote and the second saves over the first. Recovery makes this more likely, because a repeated step reads its inputs from caches and checkpoints. In the full build every quote carries a version, and a write from an out-of-date screen is rejected with HTTP 409.
Each row of the table starts with what a person watching the system would see.
| The failure | What you see | Cause | Control | Cost of the control |
|---|---|---|---|---|
| The model is slow or unavailable | Calls hang, or 5xx and 529 | Provider load or an incident | Time limit; retry with backoff; fallback model; pause the run | Waiting; different answers and a cold cache on the fallback |
| Rate limited | 429 | Your own traffic passed a limit | Obey retry-after; back off; limit concurrency | Slower runs; throughput has a ceiling |
| Malformed output | Status 200, text that fails the check | The model did not follow the contract | One repair turn, then a named failure | A full extra model call |
| A tool fails or times out | An error result, or a long wait | The system behind the tool | Time limit; retry reads in code; error text back to the model | Extra calls; an abandoned call may still be running |
| The run exceeds its budget | A growing bill, noticed late | Retries, repairs and long loops add calls | Token or dollar budget between steps | Overshoot by one call; spent work with no result |
| A side effect is repeated | Two emails | Retry, resume or redelivery after an effect, before its record | Idempotency key; outbox | A table, a worker, a key design |
| The loop never converges | Many turns, no decision | Nothing in the model's replies forces an end | Turn limit; budget; route to a person | The cut-off run is wasted |
| A dependency returns stale data | A confident, wrong quote | Caches and replays serve old facts | Versions and fetch times; reject stale writes | Rejected writes that someone must redo |
Read the last column as one list. Every control costs time, money or work that someone must redo. Reliability does not remove failure. It chooses which cost to pay, in advance and in code.
10. Cost is a reliability problem#
Cost sits in job 7 because the same events drive both. Context showed the cause: the model keeps nothing between calls, so every step sends the whole context again. Add the usage fields of each model call in a clean run of the miniature, and the totals climb.
| Model call | 1 | 2 | 3 | 4 | 5 | Whole run |
|---|---|---|---|---|---|---|
| Tokens counted for the call | 618 | 702 | 795 | 905 | 993 | 4,013 |
These are the scripted model's estimates, and the run is tiny. The shape carries over. The fifth call is about 60% larger than the first, because it resends four turns of history. An extra call late in a run is the most expensive kind. The repair turn in section 4 was one: 1,074 tokens, more than any call of the clean run.
Retries multiply the most expensive call. A repair, or a model that asks for a failed tool again, resends everything so far. The documentation read for this page does not say whether a request rejected with 429 or 529 is billed. A request that returned a response is.
A long loop costs more than its length. If every call is larger than the last, twice the turns cost more than twice as much.
Caching lowers the price and raises the ceiling. Prompt caching makes the unchanged prefix cheaper on every resend; Context has the arithmetic, including the price of writing to the cache. It also eases rate limits: the Claude API's rate limits page says the default input-token limit counts "only uncached input tokens".
For scale: the full build reports a model cost of about 0.4 cents for a quote with no flags and about 2.6 cents for a flagged one, cache writes and reads counted. The flagged quote enters a second agent loop.
11. The abstractions, each for the cause it removes#
Four functions did job 7 in the miniature. Each rung of the ladder takes some of them over. For each, ask the two questions of this chapter: does it try again for you, and does it make trying again safe?
A provider SDK retries the request#
The cause it removes: the hand-written loop of section 2 and its gaps.
The official Claude SDKs retry inside the call. The Python SDK page says: "By default, the SDK retries certain errors twice, with a short exponential backoff", and lists them: "connection errors (for example, a network problem), 408 Request Timeout, 409 Conflict, 429 Rate Limit, and >=500 Internal errors". The TypeScript and Java pages state the same default of 2, and all three give a default request timeout of 10 minutes. Errors arrive as typed exceptions: RateLimitError and InternalServerError in Python and TypeScript, RateLimitException and InternalServerException in Java.
The fragments below set both options. They are adapted from the official SDK documentation and were not run here.
Python
client = Anthropic(
max_retries=4, # default is 2; 0 turns retries off
timeout=20.0, # seconds; default is 10 minutes
)TypeScript
const client = new Anthropic({
maxRetries: 4, // default is 2
timeout: 20 * 1000, // milliseconds; default is 10 minutes
});Java
AnthropicClient client = AnthropicOkHttpClient.builder()
.fromEnv()
.maxRetries(4) // default is 2
.timeout(Duration.ofSeconds(30)) // default is 10 minutes
.build();A timeout raises APITimeoutError in Python and APIConnectionTimeoutError in TypeScript, and both pages say timeouts are retried twice by default. The Java page does not say whether a timeout is retried.
What it costs: retries at two layers multiply. Put an SDK with 2 retries inside a loop like call_with_retry with 4 attempts, and one step can send 12 requests. Retry at one layer, and turn the other off.
What stays with you: the rest of this page. The provider SDK keeps no budget across calls and never runs a tool, so the email is as unprotected as before.
An agent SDK adds limits around the loop#
The cause it removes: nobody counts across calls. An agent SDK runs the loop, so it can count turns and money and change the model.
The Claude Agent SDK documents an option for each. fallback_model is the "Fallback model to use if the primary model fails". max_turns is "Maximum agentic turns". max_budget_usd will "Stop the query when the client-side cost estimate reaches this USD value". API retries are built in: CLAUDE_CODE_MAX_RETRIES is "maximum API retries. Default 10, capped at 15".
What it costs: the cost-tracking page says the cost fields "are client-side estimates, not authoritative billing data". It is the guard of section 5, in dollars. Idempotency for your own tools remains yours.
A framework turns each control into a component#
The cause it removes: reliability code tangled into the loop. An agent framework lets you add each control as a separate piece wrapped around the model call or the tool call, usually called middleware.
LangChain's built-in list reads like sections 2 to 5 of this page: ModelRetryMiddleware ("Automatically retry failed model calls with exponential backoff"), ToolRetryMiddleware, ModelFallbackMiddleware and ModelCallLimitMiddleware ("Limit the number of model calls to prevent infinite loops or excessive costs"). Others draw the line elsewhere. The OpenAI Agents SDK says "Retries are runtime-only and opt in". Microsoft Agent Framework's documentation shows retry and fallback as middleware you write.
What it costs: a component is easy to add without thinking. A tool-retry component around send_reply is the opening scene of this chapter, installed on purpose. Frameworks has a tab per product.
A model gateway offers the same controls from beside the ladder, as one service in front of every model call. LiteLLM's routing page lists "cooldowns, fallbacks, timeouts and retries (fixed + exponential backoff)". A cooldown is the memory pick_model lacked in section 3. The full build routes every model call through a gateway of its own.
A graph runtime or a durable execution engine retries whole steps#
The cause it removes: the process itself dying. These products save progress and run a failed step again; Control and state is their home. What matters here is the contract they state about repeats.
LangGraph has a RetryPolicy per node, with a default of 3 attempts. Its fault-tolerance page is plain about what a retry is: "A retry re-runs the entire node function from the start." That "gives at-least-once semantics for node execution", and "It does not make external side effects happen exactly once." The conclusion is in the vendor's words too: "Application code must make those effects idempotent."
Temporal, which calls itself "the open source Durable Execution platform for crash-proof applications and AI agents", says the same of its unit of work, the Activity: it "may be executed multiple times", and "Temporal recommends that Activities be idempotent". Its default retry policy for Activities starts at a 1-second interval, doubles, and has no limit on attempts: "The default is unlimited."
What it costs: an unlimited retry suits a database write and is dangerous for a model call that bills on every attempt. Set the attempt limit for anything that costs money, and give every side effect a key.
A managed platform runs the retries, and names the limits#
The cause it removes: operating any of this. The vendor runs the loop and handles the transient failures of its own model calls. How much the documentation says varies. Claude Managed Agents, in beta, has a session status rescheduling: "Transient error occurred, retrying automatically." It also has a real cap: "A session budget is an optional hard spend ceiling you set when you create a session", and a session at its budget "pauses and goes idle rather than terminating". Amazon Bedrock AgentCore's Harness has three limits per invocation: maxIterations (default 75), timeoutSeconds (default 3600) and maxTokens. OpenAI's Agents API says "OpenAI manages sessions, orchestration, context compaction, and recovery", and no more about retries.
What it costs: you cannot tune the policy, and the platform retries its own calls, not yours. Foundry Agent Service's limits page tells the caller to "Implement exponential backoff with jitter in your application retry logic". Managed platforms compares them.
12. Who runs job 7#
Two questions per product: who tries again, and who limits the spend. BUILT IN means the product does it in your process, MANAGED that the vendor runs it, YOU BUILD that it stays with you.
| Product | Retry and fallback | Limits on spend |
|---|---|---|
| The miniature | YOU BUILD: call_with_retry, pick_model | YOU BUILD: over_budget, in tokens |
| Claude SDKs (Python, TypeScript, Java) | BUILT IN: 2 retries with backoff, 10-minute timeout. Fallback: YOU BUILD | YOU BUILD |
| Claude Agent SDK | BUILT IN: API retries, fallback_model | BUILT IN: max_budget_usd (an estimate), max_turns |
LangChain create_agent | BUILT IN: model retry, tool retry and model fallback middleware | BUILT IN: limits on model calls and tool calls, not on money |
| LangGraph | BUILT IN: RetryPolicy and a timeout per node | The pages read name no token or cost limit |
| Pydantic AI | BUILT IN: ModelFallback, retries | BUILT IN: UsageLimits, for example a limit on requests |
| OpenAI Agents SDK | BUILT IN: retries, opt in. No model fallback on the page read | BUILT IN: max_turns |
| CrewAI | BUILT IN: max_retry_limit, max_rpm | BUILT IN: max_iter, max_execution_time |
| Microsoft Agent Framework | YOU BUILD: the docs show retry and fallback as middleware you write | BUILT IN: max_function_calls, unlimited by default. A token or cost limit: the docs read do not say |
| Google ADK | BUILT IN: a retry policy per node in graph workflows | BUILT IN: max_llm_calls, 500 by default |
| Temporal | BUILT IN: automatic retries of Activities | NOT ITS JOB |
| LiteLLM | BUILT IN: retries, fallbacks, cooldowns, timeouts | BUILT IN: spend tracking |
| Claude Managed Agents | MANAGED: transient errors retried automatically | MANAGED: a session budget, a hard ceiling |
| Amazon Bedrock AgentCore | The pages read do not describe retries | MANAGED: maxIterations, timeoutSeconds, maxTokens |
| OpenAI Agents API | MANAGED: "recovery" of sessions; no detail on the page read. Retrying a failed request: YOU BUILD | The page read describes no budget |
| Foundry Agent Service | YOU BUILD: back off on 429 in your application | The docs read do not say |
The Claude, LangGraph, LangChain, Temporal and managed-platform rows were read on 2026-11-16; the CrewAI row and the Microsoft Agent Framework retry cell come from the baseline check of 2026-11-14. The limits of Microsoft Agent Framework and Google ADK and the Pydantic AI fallback name are as on Frameworks. This is one column of the full matrix on Build or adopt.
The table has no column for "side effects happen once", and that is its most useful finding. No documentation read for this page fills it. Where it speaks, it hands the job back, as LangGraph and Temporal did. A product can repeat your step. Only you know what your step does to the world, and what its key is.
What you can do now, and what is still open#
You can now take any failure of an agent system and ask the two questions. Can doing it again succeed? Then retry, with a growing wait, a fallback and a limit. Is doing it again safe? For a model call and a lookup, yes. For an email, only with a key that names the effect and a store that refuses to hold the name twice. In the quote example the broker gets one email after the crash, and a run that spends too much stops with a status that says why.
A repeat is safe only when something can tell that it is a repeat. Every design in this chapter makes that one fact visible.
What is still open is how anyone knows. The retried 429 left one line on the error stream. The fallback changed the model for a whole run and changed no status. The repair made one run a quarter more expensive and still ended done. All of it worked, and none of it would be noticed. To see what a run did, and whether a change made the answers better or worse, the harness has to record and measure. That is job 8, and Observability and evals is next.