All chapters
  1. Start hereAn agent is a model call in a loop
  2. Chapter 1One call: what a model is from the outside
  3. Chapter 2The loop: where a call becomes an agent
  4. Chapter 3The abstraction ladder: six rungs over one call
  5. Chapter 4The eight jobs of a harness
  6. Chapter 5Context: deciding what the model sees
  7. Chapter 6Tools: what the model may ask for, and where it runs
  8. Chapter 7MCP: one way to plug tools into any harness
  9. Chapter 8RAG: retrieval is context assembly
  10. Chapter 9Control flow and state: stopping, pausing and resuming
  11. Chapter 10Guardrails: identity, approval and the output contract
  12. Chapter 11Reliability and cost: retries, budgets and the failures that will happen
  13. Chapter 12Observability and evals: knowing what the agent did, and whether it was right
  14. Chapter 13Many agents: more loops, and what each one costs
  15. Adopt · 1SDKs: the same loop in Python, TypeScript and Java
  16. Adopt · 2Coding harnesses: a finished harness for software work
  17. Adopt · 3Frameworks and graph runtimes: the loop as a library
  18. Adopt · 4Managed platforms: someone else runs the loop
  19. Adopt · 5Build your own: a whole harness in one file
  20. DecideBuild or adopt: who runs each job?
  21. ReferenceGlossary
  22. AboutAbout this site
Chapter 12

Observability and evals: knowing what the agent did, and whether it was right

A trace shows what happened in one run. An eval says whether a change made runs better or worse. What to record, what to score, and who provides each.

An underwriter stops by the team's channel with a question. "Last Tuesday the agent recommended a quote for a house in a flood zone. 18 Gulf Shore Blvd, Naples. Why?"

Nobody can answer. The reply email went out, so the run finished. But which tools the harness ran, what they returned, which rule text was in the request that day, and how the model got from there to "quote": none of it was kept in a form anyone can read back.

A week later a second question arrives, from inside the team. Someone has changed one sentence of the system prompt so the agent explains its reasons more clearly. Is the agent now better or worse at its job?

The two questions sound alike, and they are not. The first is about one run that already happened. The second is about every run that has not happened yet.

A trace shows what happened in one run. An eval says whether a change made runs better or worse. A trace judges nothing, and an eval explains nothing. A harness needs both, and they should never be mistaken for each other.

This is job 8 of the eight: observability and evals. The chapter starts with both halves at their smallest, in the miniature, then shows what each cannot do in that form, and builds up from there.

1. Two records for two questions#

A trace is the record of one run: an entry for each model response and each tool call, written while the run executes. A person reads it afterwards.

An eval is an automated test of what the agent produces. It takes inputs whose correct result is already known, runs each one through the agent, and compares. The kept list of inputs and agreed results is called a golden set.

Diagram: Two questions, two records · a trace is about one run, an eval is about a change

Read each column top to bottom. The left column starts from one run and ends at an explanation. The right column starts from a change and ends at a comparison. One arrow joins them: a run that went wrong becomes a new case. Click any box.

  1. One run · what happened?
    1. One email, one run: A single request going through the loop: model calls, tool calls, a pause for a person, a decision.
    2. Trace: The record of that run: one entry per model response and per tool call, all carrying the same run id.
    3. An explanation of that run: Someone reads the trace afterwards and can say which step produced the decision, what it cost and how long it took.
  2. A change · better or worse?
    1. Golden set: Inputs with the answers a person has agreed are correct. It is kept in the repository and grows over time.
    2. Eval run: Every case goes through the same loop the product uses. A grader compares what happened with what was expected.
    3. A comparison of two versions: The same cases, scored for the old version and the new one. The difference is evidence about the change, not about any single run.

One arrow joins the columns: a run that went wrong, found in a trace, becomes a new case. Nothing else crosses. A good eval score does not explain last Tuesday, and a complete trace of last Tuesday does not say whether the new prompt is an improvement.

2. The trace at the bottom: one line per thing that happened#

The miniature's whole tracing code is one function, trace in examples/quote-email/harness.py. It shows the smallest useful trace: a line of JSON appended to a file for every event. A run that leaves lines behind proves the harness recorded each step in order. It does not prove the lines are enough to explain the run, and section 4 shows that they are not.

def trace(state: dict, verbose: bool, **event) -> None:
    """Job 8: one JSON line per model response and per tool call. evals.py is the second half."""
    line = json.dumps({"run_id": state["run_id"], "step": state["step"], "turn": state["turn"], **event})
    with open(TRACE_FILE, "a") as file:
        file.write(line + "\n")
    if verbose:
        print(line, file=sys.stderr)

The harness calls it in two places: model_step after every model response, and tool_step after every tool call the harness made. The model never writes a trace line. It returns text and tool requests, and the harness records what it did with them.

Here is the Naples email, run with the scripted model.

cd examples/quote-email
/opt/homebrew/bin/python3 harness.py --fake --verbose
{"run_id": "run-5ce24601", "step": 1, "turn": 1, "kind": "model", "model": "claude-opus-5-5", "stop_reason": "tool_use", "input_tokens": 61, "output_tokens": 31, "cache_read": 0, "latency_ms": 0}
{"run_id": "run-5ce24601", "step": 2, "turn": 1, "kind": "tool", "tool": "verify_address", "verdict": "allow", "is_error": false, "latency_ms": 0}
{"run_id": "run-5ce24601", "step": 3, "turn": 2, "kind": "model", "model": "claude-opus-5-5", "stop_reason": "tool_use", "input_tokens": 145, "output_tokens": 31, "cache_read": 526, "latency_ms": 0}
{"run_id": "run-5ce24601", "step": 4, "turn": 2, "kind": "tool", "tool": "lookup_property", "verdict": "allow", "is_error": false, "latency_ms": 0}
{"run_id": "run-5ce24601", "step": 5, "turn": 3, "kind": "model", "model": "claude-opus-5-5", "stop_reason": "tool_use", "input_tokens": 227, "output_tokens": 42, "cache_read": 526, "latency_ms": 0}
{"run_id": "run-5ce24601", "step": 6, "turn": 3, "kind": "tool", "tool": "check_underwriting_rules", "verdict": "allow", "is_error": false, "latency_ms": 0}
{"run_id": "run-5ce24601", "step": 7, "turn": 4, "kind": "model", "model": "claude-opus-5-5", "stop_reason": "tool_use", "input_tokens": 328, "output_tokens": 51, "cache_read": 526, "latency_ms": 0}
send_reply is waiting for a human. Continue with: --resume run-5ce24601 --approve
{
  "run_id": "run-5ce24601",
  "status": "waiting_approval",
  "decision": null,
  "reply": "waiting_approval",
  "turn": 4,
  "tokens": 3020
}

Seven lines, then the run stops. In turn 4 the model asked for send_reply, and sending an email needs a person's approval. Each line answers a small question of its own.

FieldIn the lines aboveThe question it answers
run_idrun-5ce24601Which lines belong to the same run?
step, turn1 to 7, turns 1 to 4In what order did things happen, and which tool call followed which model response?
kindmodel or toolWas this the model answering, or the harness running a tool?
modelclaude-opus-5-5Which model answered? After a fallback the name changes
stop_reasontool_useWhy did the model stop: it wants a tool, it is finished, or it was cut off?
input_tokens, output_tokens, cache_read61, 31, 0 in step 1How large was this call, and how much of it came from the prompt cache?
tool, verdict, is_errorverify_address, allow, falseWhat did the harness run, did the permission check allow it, and did it fail?
latency_ms0How long did the step take? The scripted model answers at once, so every value here is 0

Now a person approves, in a new process, perhaps a day later. The command names the run.

/opt/homebrew/bin/python3 harness.py --fake --resume run-5ce24601 --approve

The resume appends two more lines to traces.jsonl.

{"run_id": "run-5ce24601", "step": 9, "turn": 4, "kind": "tool", "tool": "send_reply", "verdict": "allow", "is_error": false, "latency_ms": 0}
{"run_id": "run-5ce24601", "step": 10, "turn": 5, "kind": "model", "model": "claude-opus-5-5", "stop_reason": "end_turn", "input_tokens": 430, "output_tokens": 37, "cache_read": 0, "latency_ms": 0}

The final status is done, the decision is quote in the high premium band, and the run's counter says "tokens": 4013. Two processes wrote nine lines, and one search for run-5ce24601 finds all of them, in order.

Notice also what is missing: there is no step 8. Hold on to that, and to the number 4013.

3. The eval at the bottom: five emails with known answers#

The second half lives in two files. golden.jsonl holds five emails, one per line. This is the first one, reformatted here for reading.

{
  "id": "naples-complete",
  "role": "underwriter",
  "email": "From: broker@coastal-brokers.example\nSubject: Quote request\n\nPlease quote homeowners for 18 Gulf Shore Blvd, Naples FL, built 1998, 2,400 sq ft.",
  "expect": {"decision": "quote", "premium_band": "high", "missing_fields": [], "reply": "sent"}
}

A case is an input plus what must come out. The other four are an email with missing facts, an unknown address, a house built in 1925, and the Naples email with an added line telling the agent to ignore its instructions.

The two functions below are the core of evals.py. run_case shows that an eval runs the real loop, not a copy: it calls harness.run, the function the product uses. grade shows the simplest grader: plain code that compares fields. A pass proves this version of the harness produced the expected result for this input, once.

def run_case(case: dict) -> dict:
    """Run one golden email through the real harness loop and return what happened."""
    db = harness.open_db(":memory:")  # a throwaway database: the outbox is not a real mailbox
    caller = harness.Caller(user="eval", role=case["role"])
    state = harness.start_run(db, case["email"], caller)
    state = harness.run(db, FakeModel(), state, approve=True)  # the eval approves, so only policy can block
    decision = state["decision"] or {}
    return {"status": state["status"], "decision": decision.get("decision"),
            "premium_band": decision.get("premium_band"), "missing_fields": decision.get("missing_fields"),
            "reply": state["reply"], "emails_sent": db.execute("SELECT COUNT(*) FROM outbox").fetchone()[0]}


def grade(expect: dict, actual: dict) -> list[str]:
    """Compare against the expected values. Returns a list of failures (empty = pass)."""
    failures = [f"{key}: expected {expect[key]!r}, got {actual[key]!r}" for key in expect if expect[key] != actual[key]]
    if actual["status"] != "done":
        failures.append(f"status: expected 'done', got {actual['status']!r}")
    if (actual["emails_sent"] == 1) != (expect["reply"] == "sent"):  # sent, but not expected to be (or the reverse)
        failures.append(f"outbox: {actual['emails_sent']} email(s) sent, expected reply {expect['reply']!r}")
    return failures
/opt/homebrew/bin/python3 evals.py
PASS  naples-complete
PASS  tampa-missing-facts
PASS  unknown-address
PASS  miami-1925
PASS  injected-instruction
score: 5/5 = 1.00

Two details are worth copying. Each case gets a throwaway database, so an email "sent" during a test never reaches a real outbox. And grade does not stop at the model's final answer: its last check counts the rows in the outbox, which is what the system did. The script exits with an error below --min-score, so an automated check can block a change.

The code that loads cases, runs them, grades them and reports is called an eval harness. It does not serve users; it runs the agent harness as its subject.

Be exact about what 5/5 means. These cases run on the scripted model, which returns the same thing every time. So this eval tests the harness: the approval gate, the policy for a suspicious email, the send-once guard. It says nothing about the quality of a real model's decisions.

4. What breaks in a trace like this one#

Go back to the underwriter. With the nine lines, the team can say: the harness ran three lookups, all allowed, none failed; the model asked to send a reply; the run ended with a quote. That is the path. The question was "why?", and here the lines run out.

Without the request, the run cannot be replayed#

The reason for the quote is in what check_underwriting_rules returned, and in the rule text the harness put in the request. The trace has neither. It says the tool ran. It does not say the tool returned "outcome": "quote" with the reason "eligible; flood zone AE", or that the request contained the rule "Flood zone AE rates in the high band", which makes such a house quotable at a higher price.

In the miniature, part of this can still be found, by accident: the checkpoint row in runs.sqlite holds the run's messages, tool results included. But a checkpoint is working state, kept so the run can resume. It does not hold the system prompt and rule text as they were on Tuesday, and if a rule document has been edited since, rebuilding the request today gives a different request.

The cause: a model's output depends on its whole request, and the harness assembled that request from sources that change. To explain a step, or to run it again, the record must hold the request as it was sent, or exact versions of everything it was built from: the prompt version, the tool definitions, the ids of the retrieved documents.

Without every token type, the cost cannot be worked out#

Add up the token fields in the nine lines: 1,191 input, 192 output and 1,578 read from the cache. That is 2,961. The run's own counter said 4,013.

The missing 1,052 tokens are cache writes. model_step counts four usage fields for the budget, and passes three of them to trace. The field cache_creation_input_tokens is counted and never recorded. In this run the scripted model reported a cache write of 526 tokens on the first call of each process.

This is a real omission in the miniature, found by adding up its own output, and it matters because token types have different prices. On the Claude API a cache write costs more than ordinary input and a cache read costs a fraction of it; Context has the multipliers. A trace that cannot tell the types apart cannot say what a run cost, and a team that cannot say what one quote costs cannot set a budget per quote.

Without one id across steps, a paused run falls apart#

The miniature gets this right. Suppose instead each process invented its own id. Tuesday's seven lines and Wednesday's two would be two unrelated stories: one that stops for no visible reason, and one that begins by sending an email. So the id is created once, saved with the run (Control and state has the checkpoint), and written on every line by every process that touches it.

The missing step 8 is the other half. Step 8 was the moment the harness asked for approval and stopped. Nothing was traced, and nothing says who approved the send on Wednesday. That fact matters more than any token count. It also belongs somewhere else.

A trace is not an audit record. A trace helps an engineer debug; it may be sampled, and it expires. Who approved what, under which version of the rules, is a business record. Keep the two in different places, with different retention.

With everything in it, the trace is a privacy problem#

The fix for the first problem is to record the request and the results. Look at what that copies. The email holds a client's home address. For an underwriter, the request also holds a rule document marked internal: "Never share the surcharge with brokers." The harness sends that document only when the caller's role may read it. A trace store is usually readable by more people than that.

So a full trace moves protected text to a place with weaker protection. The design that follows has three parts. Record structure always: ids, steps, names, counts, timings, verdicts. Record content by a stated policy: which fields, masked how, kept how long, readable by whom. Never record credentials. In the miniature the API key travels in a request header, not in the body, so logging the body does not log the key; that stops being true the moment a tool result contains a token.

Products differ on the default, so read the one you use. Section 8 shows three that go different ways.

5. What breaks in an eval like this one#

Five cases say little#

With five cases, one failure moves the score from 1.00 to 0.80. The score has five possible steps, so it cannot show a small change at all. More important is coverage: the five cases are the five situations someone thought of. A change that breaks a sixth situation passes.

Checking only the final answer accepts a right answer reached the wrong way#

Suppose a prompt change makes a real model skip check_underwriting_rules for the Naples email. It reads the rule text in its context, concludes "quote, high band" by itself, and is right. A grader that looks only at the decision passes the case. But the design was that deterministic code decides the outcome and the model reports it. On the next house the model's own reading of the rules may be wrong, and nothing will catch it.

To see a wrong path, the grader has to read the trace of the eval run as well as its result. This is where the two halves of job 8 meet: an eval can grade a trace.

One run of a case is one observation#

The scripted model gives the same output every time. A real model does not: the same case can pass on one run and fail on the next with nothing changed. Anthropic's engineering team writes: "Because model outputs vary between runs, we run multiple trials to produce more consistent results."

No setting removes this. A lower-randomness setting, where a provider offers one, is not determinism, and the API version this site shows does not accept the sampling parameters at all. So a single pass or failure is an observation about one run. It is not yet a fact about the version.

A judge nobody measured is a second unknown#

Some things code cannot grade, such as whether the reply email states the decision accurately. The usual answer is a second model call that grades the first, a judge model. It can be wrong like any other model call. If nobody has compared its grades with a person's, a failing score may mean a bad agent or a bad judge, and there is no way to tell which.

6. Building up the trace: spans, a tree, and a price on every step#

Flat lines cannot show that one thing happened inside another. A model step that took nine seconds may be one slow call, or a failed call, a wait and a second call to a fallback model. To show that, each record gets a start time, an end time, and the id of the record it happened inside. Such a record is called a span. OpenTelemetry, a widely used open standard for tracing, defines it in one line: "A span represents a unit of work or operation." A span names its parent, and "child spans represent sub-operations". All spans of one run share a trace id, so the run becomes a tree.

Diagram: One run as a tree of spans · illustrative, for a run where the first model attempt failed

Read left to right. The run is the root. Each step is a child of the run. Each attempt to reach the model is a child of its step. Every box carries the same run id and the id of its parent. The miniature writes flat lines; this is the shape a tracing standard gives the same facts. Click any box.

  1. 1 · The run
    1. Run run-5ce24601: The root span. It starts when the email arrives and ends at the decision, however long the pause for a person lasts.
  2. 2 · Steps
    1. Model turn 3: One model response inside the run: stop reason, the tool it asked for, tokens of each type.
    2. Tool call check_underwriting_rules: One tool call the harness made: which tool, the verdict of the permission check, error or not, how long it took.
    3. Wait for approval: The run is paused. A span with a start, an end and the name of the person who approved makes the pause visible.
  3. 3 · Attempts
    1. Attempt 1 HTTP 529: One HTTP request to the model provider that failed. Without this level a retry is invisible: the step only looks slow.
    2. Attempt 2 fallback model: The attempt that succeeded, with the model that answered, its latency, its token counts and the provider's request id.

The tree is illustrative; the miniature writes flat lines. The run is the root and lasts from the email to the decision, pause included. Each model turn, tool call and wait is a child of the run. Each HTTP attempt is a child of its turn, which is where a retry and a fallback from Reliability and cost become visible. The wait for approval is a span too, so the missing step 8 has a place.

What each span carries follows from section 4.

SpanWhat it must carryThe problem from section 4 it removes
RunRun id, caller's role, prompt version, final status, totalsLines that cannot be joined; a request that cannot be rebuilt
Model turnStop reason, tool requests, all four usage fieldsA path without its reasons
HTTP attemptModel, status, latency, usage, the provider's request idCost that cannot be worked out; an invisible retry
Tool callTool, verdict, error flag, latency; arguments and result under the content policy"The tool ran" without what it returned
Wait for a personStart, end, a reference to the approval recordA pause that looks like a gap

Two notes on cost. Cost is tokens of each type multiplied by a price list, and price lists change, so store the token counts and the model as facts and treat the money figure as derived. And a figure computed in your process is an estimate: the Claude Agent SDK says its own cost fields "are client-side estimates, not authoritative billing data". The provider's request id ties a span to the provider's side; on the Claude API "every API response includes a unique request-id header".

In the full build: four levels of records and one idQuoted from the source article; not re-measured

The full build of the quote system records four levels. A run is one quote going through one model step: stage, model, prompt version, turns, tokens, cost, outcome. A turn is one model response. A call is one HTTP attempt to a model or a data service: attempt number, status, latency, cost. The wire level is the full request and response as JSON; it contains customer data, and the production profile turns it off.

One id goes through everything: the quote id plus the stage is on every row and on the log lines of all three processes. Human decisions are kept in their own table, as business records.

These are tables in the system's own database, so one SQL query can join a trace with business rows, such as cost per stage. The price is that every screen, alert and eval runner is the team's to build.

7. Building up the eval: more cases, graded by code first#

Where golden cases come from#

A golden set grows from four sources. Real runs that went wrong: last Tuesday's quote, once understood, becomes a case with the answer the underwriter says is right. The business logic: one case per rule and branch, such as the 1950 cut-off and each flood zone. Hostile input, like the injected instruction. And the cases people argued about, because a case two underwriters disagree on is a hole in the specification.

The expected answers are part of the specification, and the person who owns the decision reviews them the way code is reviewed. On size, Anthropic's guidance for a first set is modest: "20-50 simple tasks drawn from real failures is a great start."

Grade with code wherever code can decide#

Code is the first grader to reach for: it is fast, costs nothing per run, and gives the same grade every time. grade already checks the decision, the band, the missing fields and the outbox. The next step is to check the path, using the trace lines of the eval run.

The sketch below does not demand an exact order of tool calls. The same Anthropic article calls a check on "a sequence of tool calls in the right order" too rigid, because a good agent may find another valid path. The sketch checks two things that must hold on every valid path.

# sketch: a path check over the trace lines of one eval run (illustrative; not part of the miniature)
def path_failures(lines: list[dict], decision: str) -> list[str]:
    tools = [line for line in lines if line["kind"] == "tool"]
    ran = [line["tool"] for line in tools if line["verdict"] == "allow" and not line["is_error"]]
    failures = []
    if decision == "quote" and "check_underwriting_rules" not in ran:
        failures.append("quoted without running the underwriting rules")
    if any(line["verdict"].startswith("denied") for line in tools):
        failures.append("the model asked for a tool this role may not run")
    return failures

The first check catches the right answer by the wrong path from section 5. The second catches what a final answer never shows: the model asked for a tool and the harness refused.

A judge model, only where code cannot decide, and only after it is measured#

What is left for a judge is small on purpose: the wording of the reply, and whether the stated reasons match the tool results in meaning. A judge works from a rubric, written scoring rules such as "the reply states the premium band and never mentions the internal surcharge".

Before a judge's grade may block a change, measure the judge: run it on outputs people have already graded, and compare. An illustrative result: underwriters graded 100 replies, and the judge agreed on 91. That number hides the important part. If 7 of the 9 disagreements are replies the judge passed and the people failed, the judge is lenient in the direction that lets bad replies through. Such a judge can advise. It should not gate.

Anthropic's article says the same: "LLM-as-judge graders should be closely calibrated with human experts".

With a real model, decide in advance what a difference means#

Against a real model, run each case more than once, and keep three statements apart. "The new prompt failed the 1925 case in all three runs, and the old one passed it in all three" is an observation, and a strong signal. "This case passes on some runs and fails on others" shows the system varies; it does not say by how much. "The new version is better" is a claim about the versions. It needs enough runs, a measured baseline of how much scores move when nothing changes, and a stated method. Three runs rarely support it.

What a team can do without statistics is set a trigger before looking at results: for example, any case that must never fail and now fails once goes to a person. The full build runs the same cases in two modes: against the real model to grade the model and prompt, at about 2 cents for a flagged quote in that build, and against a scripted model to test the harness for free.

Before the change and after it#

An eval on a golden set runs before a change is released: an offline eval. After release there is a different job, online monitoring. LangSmith's documentation states the difference in one phrase: online evaluation targets "real production traces without reference outputs".

Without a reference output, less can be scored: rules that hold on every run (the path checks above, the rate of repaired outputs, cost and latency per quote), a judge or a person on a sample, and, in this example, how often the underwriter overrides the recommendation. A person decides every quote anyway, so each override is a graded example, and the next golden case.

How to evaluate a change to an agent inside a delivery pipeline, quickly enough to do on every change, is the subject of a sister publication: Changing an agent safely.

8. The abstractions: a standard, a product, or built in#

Everything so far can be built by hand, and the full build did. Three kinds of abstraction take part of it over.

A tracing standard, so the record outlives the tool that reads it#

The cause it removes: a home-made trace format can only be read by home-made screens. OpenTelemetry defines spans, the trace tree and a wire protocol that many products can receive. An exporter is the piece that sends spans from your process to one of them. Changing the product means changing the exporter, not the harness.

For model calls, OpenTelemetry has the GenAI semantic conventions: agreed attribute names such as gen_ai.operation.name, gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, and operation names such as chat, invoke_agent and execute_tool.

What it costs: the conventions carry the status "Development", so names can still change. They do not know your business: the quote id, the stage and the prompt version are attributes you add. On privacy they take a position: attributes that hold message content, such as gen_ai.input.messages, are marked "Opt-In", with the note "This attribute is likely to contain sensitive information including user/PII data."

A tracing and eval product, so the screens are not yours to build#

The cause it removes: a trace nobody can browse is a log file. A trace viewer, search, dashboards, alerts, dataset storage and an eval runner are a product of their own.

LangSmith is one such product. For LangChain and LangGraph code its documentation says tracing is enabled "with a single environment variable" (LANGSMITH_TRACING=true, with LANGSMITH_API_KEY); other code wraps its model client with wrap_anthropic and marks functions with @traceable. Its evaluators are code, judge, human and pairwise. Langfuse and Arize Phoenix are open source; Phoenix says it "is built on top of OpenTelemetry". Pydantic Evals is an evaluation library built around a Dataset of Case objects and Evaluator classes.

What it costs: a vendor and a price, or a service to host. Traces contain prompts, so where they are stored is a decision; LangSmith offers cloud, hybrid and self-hosted forms. Joining traces with business data needs an export. And some things stay yours whatever you adopt: the golden cases, the path checks, redaction before a trace leaves your network, the business records, and one id across the parts that are not model calls.

Tracing built into the SDK or the platform#

The cause it removes: instrumenting a loop you did not write. When an agent SDK or a managed platform runs the loop, only it sees each step, so it has to emit the trace.

Here are the three defaults promised in section 4. The Claude Agent SDK emits OpenTelemetry, but "telemetry is off until you set CLAUDE_CODE_ENABLE_TELEMETRY=1 and choose at least one exporter", its tracing "is in beta", and content is left out unless you opt in: "The content your agent reads and writes is not recorded by default." The OpenAI Agents SDK goes the other way on both: "Tracing is enabled by default", and "by default, trace_include_sensitive_data is True". The Claude API's own OpenTelemetry export, also in beta, records metadata only by default, with full content as a setting.

Claude Managed Agents runs the loop on the vendor's side, so the trace is the vendor's too: a session viewer in the Claude Console, in beta, to "inspect what an agent did in a session, without writing any code", and a usage field on the session with token counts and a list cost.

What it costs: you get the spans the vendor chose, under the vendor's names, and only for the agent loop. The email intake before it and the approval record after it are still yours to connect. And the pages read for this chapter describe traces, not a golden set: evals stay with you or with a separate product.

9. Who runs job 8#

The table keeps the chapter's split: traces in one column, evals in the other. BUILT IN means the product does it in your process or deployment, INTEGRATES that it connects to another product for it, MANAGED that the vendor runs it, YOU BUILD that it stays with you. Rows were read on the vendors' documentation on 2026-11-14 or 2026-11-16.

ProductTracesEvals
The miniatureYOU BUILD: trace, flat JSON linesYOU BUILD: evals.py, five cases, graded by code
Claude Messages APIUsage numbers and a request-id per response; OpenTelemetry export in beta. The rest is YOU BUILDYOU BUILD
Claude Agent SDKBUILT IN: OpenTelemetry, off until enabled; traces in beta; cost fields are estimatesThe docs do not say
Claude Managed AgentsMANAGED: Console session viewer (beta), session usage with list cost, OpenTelemetry tracingThe observability page does not mention evals
OpenAI Agents SDKBUILT IN: tracing, on by defaultThe tracing page does not mention evals
LangChain create_agent, LangGraphINTEGRATES: LangSmithINTEGRATES: LangSmith
LangSmithBUILT IN: tracing, dashboards, alertsBUILT IN: offline and online evaluation
LangfuseBUILT IN: tracingBUILT IN: evals, datasets
Arize PhoenixBUILT IN: tracingBUILT IN: evals, datasets and experiments
Pydantic AIBUILT IN: OpenTelemetry. INTEGRATES: LogfireBUILT IN: Pydantic Evals
Google ADKBUILT IN: logging, metrics, tracesBUILT IN: evaluation
Microsoft Agent FrameworkBUILT IN: OpenTelemetryThe pages read do not say
Amazon Bedrock AgentCoreMANAGED: ObservabilityMANAGED: Evaluations
Foundry Agent ServiceMANAGED: tracing, metricsMANAGED: evaluations
Google Agent RuntimeMANAGED: observability, trace viewerMANAGED: agent evaluation

Read the table by column. Every product here that runs a loop offers some trace, so the trace is the half that is easier to adopt. Even where the eval column says BUILT IN or MANAGED, the product supplies the runner and the screens. The cases, the expected answers and the decision about what a failure means are in no product. Build or adopt has the full map of products and jobs.

What you can do now, and what is still open#

With the records of section 6, the team can answer the underwriter: the rules tool returned a quote in the high band for flood zone AE, this rule text was in the request, this person approved, and the run cost this much. And when the next prompt change arrives, a golden set graded mostly by code says which cases moved, and calls three runs an observation.

Everything here assumed one loop with one run id. The full quote system has two agent loops and two single model calls, each with its own context. Once one agent hands work to another, a trace has to follow the work across loops, and an eval has to decide which loop a failure belongs to. Why a system is split into several agents, and what each added loop costs, is the subject of Many agents.

Record enough to explain any single run, and keep enough cases to compare any two versions. The first is a trace and the second is an eval. Most of what goes wrong in job 8 comes from asking one of them the other's question.