Your agent worked fine in the demo. Three weeks into production, a user reports it "just hangs sometimes," your OpenAI bill doubled, and you have no idea which of the 14 LLM calls in your pipeline is responsible. Traditional APM tools show you a healthy green dashboard the whole time.
That gap is what LLM observability exists to close. Standard monitoring tells you the HTTP request to your API returned 200 in 45 seconds. It cannot tell you that the model retried a malformed tool call four times, burned 60,000 tokens re-reading the same context, and finally answered a question the user never asked.
Why Normal Monitoring Falls Short for LLM Apps
Classic observability assumes deterministic code. Same input, same output, same latency profile. LLM pipelines break every one of those assumptions.
A single agent request might fan out into a planning call, three tool calls, a retrieval step, and a synthesis call. Any of those can silently degrade: the model picks the wrong tool, the retrieved context is stale, or a prompt change last Tuesday made summaries 40% longer and quietly doubled output token costs.
None of that shows up as an error. The request succeeds. The output is just worse, slower, or more expensive than it should be. You need visibility into the semantic layer, not just the transport layer.
The Four Things Worth Tracing
After the logging basics, almost everything useful in LLM observability falls into four buckets.
1. Prompts and completions
Capture the exact rendered prompt sent to the model, not the template. Template bugs (a variable that renders as None, a system prompt that got truncated) are among the most common production failures, and you can only catch them by looking at the real payload.
Store completions alongside them. When a user complains about a bad answer, you want to replay the exact input that produced it.
2. Tool calls
For agents, tool calls are where most failures live. Trace each call as its own span: which tool, what arguments the model generated, what came back, and how long it took.
Malformed arguments are the classic failure. The model calls fetch_page(url="the pricing page") instead of passing an actual URL, the tool errors, the model retries, and your latency triples. Without per-tool-call spans you see "slow request." With them you see exactly which tool and which argument pattern fails.
Web-facing tools deserve extra attention because they are the least deterministic part of the stack. A fetch that returns 500KB of navigation boilerplate instead of article text will poison every downstream call in the trace. If your agent touches the web, log the response size and format of every fetch. (This is one reason we built link.sc to return clean Markdown: smaller tool outputs make traces cheaper and failures easier to spot. More on that in our post on giving agents internet access.)
3. Cost
Track tokens per span, then roll them up per trace, per user, and per feature. The roll-ups matter more than the raw numbers. "This customer's requests average 90,000 tokens because their documents blow up the context window" is actionable. A monthly invoice is not.
Tag traces with model name and version. When you swap models or a provider changes pricing, you want cost deltas per pipeline stage, not a single blended number.
4. Latency
Record time-to-first-token and total generation time separately. They fail differently: TTFT spikes usually mean provider queueing or oversized prompts, while long total times point to runaway output length or retry loops.
In agent pipelines, also watch the gaps between spans. I have seen traces where the LLM calls totaled 8 seconds but the request took 30, and the missing 22 seconds were sequential tool calls that could have run in parallel.
Langfuse vs LangSmith vs OpenTelemetry
The three names you will hear most, and they are not interchangeable.
| Langfuse | LangSmith | OpenTelemetry | |
|---|---|---|---|
| What it is | Open source LLM observability platform | Managed platform from the LangChain team | Vendor-neutral instrumentation standard |
| Self-hostable | Yes (MIT-licensed core) | Enterprise plans only | N/A (it is a spec plus SDKs) |
| Best fit | Teams wanting control over trace data | Teams already deep in LangChain/LangGraph | Teams with existing OTel infra (Grafana, Datadog, Honeycomb) |
| Prompt management | Built in | Built in | Not included |
| Evals | Built in | Built in | Not included |
| Lock-in risk | Low | Moderate | Lowest |
The practical distinction: Langfuse and LangSmith are products with UIs, prompt versioning, and eval tooling. OpenTelemetry is plumbing. Its GenAI semantic conventions define standard attribute names like gen_ai.usage.input_tokens and gen_ai.request.model, so any OTel-compatible backend can understand your LLM spans.
These are converging, not competing. Langfuse accepts OTel traces natively, and LangSmith added OTel ingestion too. Instrumenting with OTel conventions and picking your backend later is a defensible default in 2026.
What Instrumentation Actually Looks Like
Here is a minimal Langfuse example tracing an agent step with a web fetch tool:
from langfuse import observe, get_client
@observe(as_type="tool")
def fetch_page(url: str) -> str:
# link.sc returns clean Markdown, so the span
# output stays small enough to inspect in the UI
resp = requests.get(
"https://api.link.sc/fetch",
params={"url": url, "format": "markdown"},
headers={"Authorization": f"Bearer {API_KEY}"},
)
return resp.json()["content"]
@observe()
def answer_question(question: str) -> str:
context = fetch_page(find_source(question))
return llm_call(question, context)
Two decorators and you get nested spans with inputs, outputs, timings, and token counts attached. The OTel equivalent is a few more lines but follows the same shape: one span per LLM call, one per tool call, all under a single trace ID.
Whatever you use, propagate a trace ID from the user request down through every LLM and tool call. Orphaned spans are the number one reason LLM traces end up useless.
Start Smaller Than You Think
The mistake I see teams make is trying to build evals, drift detection, and cost dashboards on day one. Skip all of that initially.
Week one: trace every LLM call and tool call with inputs, outputs, tokens, and timing. That alone answers 80% of production questions.
Week two: add user feedback signals (thumbs up/down, regenerate clicks) as scores on traces. Now you can filter for bad traces instead of sampling randomly.
Later: turn recurring failure patterns into automated evals, and set alerts on cost per trace and TTFT percentiles.
One last observation from our own traces: the noisiest spans are almost always web tool outputs. Feeding raw HTML into a model bloats both your token bill and your observability storage, since every trace carries those payloads. Cleaning tool outputs before they hit the model (see our token optimization guide) is the rare fix that improves cost, latency, and trace readability at the same time.
Building an agent that needs clean, traceable web data? Get a free link.sc API key and keep your tool call spans small.