Your agent said it worked. The trace says otherwise.
tracelint is a structural linter for tool-calling agents. Capture a run in your tests, lint the trace, and fail the build on the defects a fluent final answer hides — deterministically, with no model in the loop.
get_order(order_id='A100')R2arefund_order(order_id='A100')R2brefund_order(order_id='A100')R8get_order returned an error (HTTP 500), read straight off the span.
The errored value A100 was reused as an argument to the side-effecting refund_order — no fallback.
refund_order ran twice with identical args — the card was refunded twice.
Langfuse
Phoenix
OpenAI
LangSmith
OpenTelemetry
Why it exists
The bugs that never reach the answer.
A fluent final message passes your evals and your LLM judge — while the run underneath already went wrong. These defects are structural, and decidable straight from the trace:
A deploy tool returns a 500 Internal Server Error, but the agent swallows it and reports the deployment as successful.
A payment API returns {"status":"declined"} over an HTTP 200, so the exception path never fires and the agent proceeds as if it succeeded.
A non-idempotent send_email is invoked twice due to a retry, but the agent's final answer still looks correct.
Different tools, one kind of bug: the failure is in what the agent did, not what it said — so it's provable straight from the trace. That's the part tracelint checks.
How it works
It runs after the run, on the trace — never on a model.
Many agent failures aren't a matter of taste; they're decidable from what the agent actually did. tracelint decides them, the same way every time.
Capture a run, or read yours
Capture a run in a test, or read the spans you already collect — Phoenix, Langfuse, LangSmith, OpenAI, OTel. Everything normalizes into one canonical schema: no rewriting your agent, no SDK to embed. If you can export the trace, tracelint can read it.
with capture("trace.json", framework="smolagents"): ...
Replay the run, deterministically
Structural checks read what the agent actually did — schema violations, ignored tool errors, reused failed values, loops, duplicate side effects — each pinned to the exact steps as evidence. No model in the loop, so the same trace always yields the same report.
lint_trace(trace, default_rules())
Get a verdict you can act on
A CI exit code that fails the build on a hard defect, with every finding backed by its evidence — never a silent pass. Emit JSON or SARIF for code scanning, and read the coverage to see how much of the run was actually verified.
$ tracelint check spans.json → fails CI
No prompt, no second model, no stochastic verdict — the same trace always gives the same report.
Only structurally-provable defects fail CI; heuristics are shown as candidates with their evidence.
If a check can't run — a field it needs is missing — it says so, with the reason. A clean report means everything was actually checked.
Per-rule "evaluatable / total" coverage says how much of the run was actually verified — not just that nothing fired.
It owns the structurally-decidable part before the probabilistic interpretation — alongside your LLM-judge evals, not instead of them.
The checks
What it catches
Deterministic checks across schema, error handling, provenance, control flow, and side effects. Every finding is filed under one of three tiers — and only the first one fails your build:
Provable from the trace itself. The run fails CI — the build stops.
A certain fact — a tool errored, a side effect repeated. Reported; it doesn't fail CI.
A heuristic, shown with its evidence for you to judge. Never fails CI on its own.
The rules — grouped · hover a rule id for its definition
Invalid tool call
Arguments that don't satisfy the tool's JSON Schema, aren't well-formed, or a call to a tool that isn't in the declared toolset.
Failed operation
A tool returned a structured error — and, worse, its value got reused by a later side-effecting call instead of being handled.
Unexplained value
An argument value that isn't derivable from anything the agent actually observed in the trace.
No-progress execution
The same call repeated with no change in state, or an identical call and result with no mutation in between (retries and polls excluded).
Duplicate side effect
A non-idempotent side-effecting call repeated with equivalent args after the first succeeded — a double charge, refund, or email.
Get started
Capture → lint → CI.
The core loop is three steps: capture a run in your tests, lint the trace, and fail the build on provable defects. Most rules run keyless; add a tools.json to declare the tool contract (side_effecting, failure_when, x-value-origin) and light up the schema- and side-effect checks. Validated on real traces from smolagents, LangGraph, and CrewAI.
"tracelint[capture-smolagents]" to capture a run in a test (also capture-langchain, capture-crewai)
1 · Capture a run
No trace file yet? capture records one by wrapping your framework's own OpenInference instrumentation — your live tracing stays untouched.
from tracelint.capture import capture with capture("trace.json", framework="smolagents"): agent.run("...")
2 · Lint it, gate CI
Lint the captured trace. A hard defect exits non-zero, so it fails the build like a failing test.
In-process capture covers smolagents, LangGraph, and CrewAI. Langflow is a server — export its OTel trace to a file, then run the same check.
Already have traces? Point tracelint at any source:
Langfuse
Pull a specific trace to lint or reproduce it — or export any trace to a file and check it.
Then tracelint check trace.json. Writing findings back into Langfuse as Scores stays available as an advanced, opt-in recipe.
Arize Phoenix
Lint the OpenInference spans Phoenix already collects — no new instrumentation.
Reads get_spans_dataframe() records directly, too.
OpenTelemetry
Any OTLP source — the event-list reader understands the OTel GenAI semantic convention (OpenLLMetry / Traceloop).
LangSmith
Nested LangSmith run trees, normalized into the canonical schema.
OpenAI
Chat-completion message lists, including the ShareGPT shape.
Custom format
Map any shape to the canonical Trace in a few lines — every rule applies.
Tool contract tools.json
Tell tracelint the semantics a trace can't infer — declared once, never guessed from a tool's name.
Bootstrap it from a trace — init discovers the tools and their arg schemas; you fill only the behavior. Start without one, add only what the trace can't know.
{
"tools": {
"refund_order": {
"metadata": {
"side_effecting": true,
"idempotent": false,
"failure_when": { "pointer": "/refunded", "equals": false }
}
}
}
}
In your pipeline
Gate the build on it.
Add it in a few lines. A hard defect fails the run — the same way a failing test does.
# .github/workflows/ci.yml name: tracelint on: [push, pull_request] jobs: tracelint: runs-on: ubuntu-latest steps: - uses: AshwinUgale/tracelint@v0.8.0 with: path: spans.json format: openinference tools: tools.json
$ tracelint check trace.json --format openinference --tools tools.json
No hard defects — findings and coverage still reported.
A structurally-provable defect. The build fails.
A bad trace or tools file — never a false pass.
Upload to GitHub, GitLab, or Azure DevOps code scanning for inline annotations.