LLM Evaluations

OpenObserve offers two evaluation modes: Online Evaluations, which score live traces and spans as they happen, and Experiments, which run scorers over a static dataset for offline testing and comparison. Both use LLM-as-a-judge scorers (your own AI provider) or external remote scoring endpoints.

Overview

Activating an eval job runs your scorers against incoming data. Scores land in the _llm_scores stream; evaluator telemetry lands in the _evaluator stream.

Enable Online Evaluations

Online Evaluations is enterprise-only, enabled by default:

O2_ONLINE_EVALS_ENABLED=true

When disabled, the Evaluations navigation is hidden.

Trace- and session-scope jobs are picked up by a background scheduler that polls trace streams (default every 45 seconds):

O2_EVAL_SCHEDULER_POLL_INTERVAL_SECS=45

Providers

A Provider stores the connection details for an LLM API. Navigate to Evaluations > Providers > Add Provider.

Use openai_compatible for any OpenAI-chat-completions-compatible service (supply the full endpoint URL) and vllm for a self-hosted vLLM server (defaults to http://localhost:8000/v1/chat/completions). Both work without an API key for keyless deployments.

From the provider detail page you can Test connectivity, Update fields (leave the API key blank to keep the existing one), or Delete it; scorers using a deleted provider fail until reassigned.

Score Configs

A Score Config defines what a score looks like and when it's healthy. Navigate to Evaluations > Score Configs > Add Score Config.

Updating a score config creates a new version; scorers can pin to a specific version or always use the latest.

Scorers

A Scorer is the evaluation unit: a prompt template with {{variable}} placeholders, execution parameters, and an optional linked score config. Navigate to Evaluations > Scorers > Add Scorer.

• LLM Judge: sends the rendered template to an LLM via a provider. Configure provider, model, temperature, max tokens, timeout, and optional reasoning.
• Remote: sends the rendered template as an HTTP request to an external service. Configure endpoint, method, auth (none/bearer/basic/api_key), headers, timeout, and retries.

Use the Test button on the scorer detail page to try it with sample variable values; you'll see the score, reasoning, model, latency, and token usage.

Like score configs, scorers are versioned. Eval jobs can reference a scorer by entity ID (always latest) or pin a specific version.

Eval Jobs

An Eval Job runs scorers against incoming traces at a chosen target scope: span, trace, or session. Navigate to Evaluations > Eval Jobs > Add Job.

Target scope

Trace and session completion

For trace- and session-scope jobs, configure how OpenObserve decides a target is "done":

Example end signal (evaluate as soon as a span with status = "complete" arrives):

{
  "version": 2,
  "conditions": {
    "filterType": "group",
    "logicalOperator": "AND",
    "conditions": [{
      "filterType": "condition",
      "column": "status",
      "operator": "=",
      "value": "complete",
      "logicalOperator": "AND"
    }]
  }
}

Span selectors (trace scope)

For trace-scope jobs, a scorer may only need a subset of a trace's spans. A Span Selector filters, picks, and limits which spans (and fields) get sent to a given scorer; bind one to any scorer whose template uses {{ spans }}.

Input mapping and template variables

Each scorer's {{variables}} are mapped to a source in the job form's Prompt variables section, either a system-provided value or a span attribute.

For trace/session scope, OpenObserve auto-supplies these as ready-to-use template variables:

For span-scope jobs, every variable maps to a span attribute instead (e.g. input → {{gen_ai_input_messages}}).

Manual evaluation

Trigger an evaluation on demand for a specific trace, span, or session. This is useful for re-testing after changing scorers. From the trace, span, or session detail page, click Evaluate trace / Evaluate span / Evaluate session, then pick which Eval Job to run.

Job lifecycle

draft → active ⇄ paused
          ↓
       degraded → active
          ↓
       archived

• Activate: starts scoring.
• Pause: stops it temporarily.
• Resume: restarts it.
• Archive: stops it permanently.

Editing an active job's config (filters, sampling, scorers, scope) takes effect immediately. No need to pause first.

Where scores go

Scores land in the _llm_scores stream; evaluator telemetry (latency, tokens, status) lands in the _evaluator traces stream. Query both directly for debugging or dashboards.

Quality Dashboard

The Quality tab gives a real-time overview of evaluation health across your score configs, agents, and streams. Drill into a score config to see KPI cards, trend charts, and a runs table, filterable by scope (All/Span/Trace/Session).

When a scorer fails, click the Scorer Failures KPI card to jump to the failing evaluator traces.

Evaluation runs

The Evaluation Runs table lists individual score records: value, health, target identity, agent, and reasoning. Click a row to open its evaluator trace. Filter to all runs or unhealthy only.

Gen-AI Agents

OpenObserve auto-discovers agents from your trace telemetry (via gen_ai.agent.name/id, or your own configured attribute fields), so you can scope the Quality, LLM Insights, and Sessions views to a single agent or All Agents.

If your telemetry uses non-standard attribute names, map them under Settings > GenAI Agent Mapping (Agent Name/ID/Environment/Version fields). Scorer templates can also reference {{agent_name}}, {{agent_id}}, and {{pipeline_source_stream_type}}.

Experiments

For offline, batch evaluation against a versioned dataset (pinned scorers, trial counts, cost estimates, baseline comparisons), see LLM Experiments.

RBAC

Providers, Score Configs, Scorers, and Eval Jobs each have their own permissions (GET, LIST, POST, PUT, DELETE). Assign roles in Identity & Access Management > Roles to control access.

API Reference

All endpoints are prefixed with /api/{org_id}.