# What is LLM Observability?

> LLM observability is the practice of monitoring LLM-powered applications - tracing prompts and responses, tracking token costs and latency, and evaluating output quality - to keep AI features reliable, safe, and affordable.

Source: https://openobserve.ai/glossary/what-is-llm-observability/
Published: 2026-07-08
Term: LLM Observability
Category: AI Observability
Related terms: what-is-observability, what-is-distributed-tracing, what-is-telemetry

---

**LLM observability** is observability for applications built on large language models: capturing the prompts, responses, tool calls, token counts, latencies, and quality signals of every model interaction so teams can debug, optimize, and trust their AI features. It extends [distributed tracing](/glossary/what-is-distributed-tracing/) to a world where the "service" is probabilistic and every call has a price tag.

## Why LLM apps need their own observability

LLM-powered systems break the assumptions of classic monitoring:

- **Non-determinism** - the same prompt can produce different outputs; "correct" is a spectrum, not a status code
- **New failure modes** - hallucinations, refusals, prompt injection, context overflow - none of which throw exceptions
- **Per-request economics** - cost scales with tokens, so one chatty prompt template or runaway agent loop becomes a real bill
- **Pipeline complexity** - RAG retrieval, tool-calling agents, and multi-model chains mean one user question triggers a tree of model and non-model operations

## What to capture

1. **Traces** - every model call as a span: model, prompt, response, token counts, tool invocations, retrieval steps; agent frameworks produce deep trees worth tracing end-to-end
2. **Cost & usage** - tokens and dollars per request, feature, tenant, and model, with alerts on anomalies (see [monitoring OpenAI API costs](/blog/monitor-openai-api-costs-opentelemetry/))
3. **Performance** - latency and time-to-first-token, provider error and rate-limit rates
4. **Quality & safety** - evaluation scores, user feedback, guardrail triggers, [PII redaction](/blog/redact-pii-llm-telemetry/) of captured prompts

## The open-standards approach

OpenTelemetry's **generative AI semantic conventions** standardize LLM telemetry, and libraries like OpenLIT, OpenLLMetry, and framework integrations ([LangChain, LlamaIndex](/blog/langchain-llamaindex-openobserve/)) emit model calls as ordinary OTLP traces. That keeps LLM telemetry in the same platform as the rest of your stack - an AI feature's slowness might be the vector database, not the model, and only correlated telemetry shows that.

## LLM observability in OpenObserve

OpenObserve provides [LLM observability](/llm-observability/) on OpenTelemetry standards, with integrations for major model providers and agent frameworks, unified with your [logs](/logs/), [metrics](/metrics/), and [traces](/traces/). See [LLM monitoring best practices](/blog/llm-monitoring-best-practices/) and the [tools comparison](/blog/llm-observability-tools/) to go deeper.