LLM Evaluations

Online Evaluations let you continuously score your LLM application's traces and spans using configurable evaluators - either LLM-as-a-judge powered by your own AI providers, or external remote scoring endpoints.

Overview

The Online Evaluations system has four core resources, each building on the previous:

When you activate an eval job, the system creates a system-managed evaluation pipeline that runs your scorers against incoming data. Scores flow into the _llm_scores stream; evaluator telemetry flows into the _evaluator traces stream.

Enable Online Evaluations

Online Evaluations is an enterprise feature. Set the configuration flag to enable it:

ZO_ONLINE_EVALS_ENABLED=true

When enabled, the Evaluations top-level navigation appears in the UI. When disabled, all evaluation pages and settings are hidden; backend API endpoints remain reachable.

Providers

A Provider stores the connection details for an LLM API. You configure one provider per AI service you want your LLM Judge scorers to use.

Create a provider

Navigate to Evaluations > Providers and click Add Provider.

Test a provider

From the provider detail page, use the Test button to verify connectivity. The system sends a test request using the configured endpoint and credentials.

Manage providers

• Update: Edit any field. The provider is updated in-place.
• Delete: Removes the provider. Scorers referencing a deleted provider will fail until reassigned.

Score Configs

A Score Config describes what a score looks like and when it is considered healthy.

Create a score config

Navigate to Evaluations > Score Configs and click Add Score Config.

Versioning

Score configs are versioned. Each config has a stable entity ID that stays the same across versions, and a unique ID per version. Updating a score config creates a new version and bumps the version number. Scorers that reference a score config can pin to a specific version or always use the latest.

Scorers

A Scorer is the executable evaluation unit. It contains a prompt template with {{variable}} placeholders, execution parameters, and an optional link to a score config that describes its output.

Scorer types

• LLM Judge: Sends the rendered template to an LLM via a configured provider. Supports temperature, max tokens, timeout, output parsing, and optional reasoning.
• Remote: Sends the rendered template as an HTTP request to an external evaluation service. Supports bearer token auth, API keys, basic auth, custom headers, timeouts, and retries.

Create a scorer

Navigate to Evaluations > Scorers and click Add Scorer.

For LLM Judge, you also configure:

For Remote, you configure:

Test a scorer

From the scorer detail page, use the Test button. Provide values for the template variables, and the system executes a one-off evaluation. The response shows the score, reasoning, model used, latency, and token usage.

Preview output schema

For LLM Judge scorers, the Preview Schema endpoint shows the derived output schema based on the score config and extra metadata fields, helping you understand what structure the LLM will return.

Versioning

Like score configs, scorers are versioned. Each scorer has a stable entity ID and a version number. Updating a scorer creates a new version. Eval jobs can reference a scorer by entity ID (always latest) or pin to a specific version.

Eval Jobs

An Eval Job is the execution unit that runs scorers against incoming traces. Each job defines a target scope — the granularity at which scoring runs: individual spans, entire traces, or full sessions.

Create a job

Navigate to Evaluations > Eval Jobs and click Add Job.

Target scope

The Target Scope determines what unit of evaluation the job scores:

Only span-scope jobs create a hidden evaluation pipeline. Trace and session jobs are detected by the Eval Scheduler, which polls trace streams for completed targets.

Trace and session completion

For trace-scope and session-scope jobs, the system must determine when a target is "complete" and ready for scoring. You control this with completion configuration:

The end signal is a standard condition expression, using the same filter syntax as the job's main filter condition. For example, to mark a trace complete when a span with status = "complete" arrives:

{
  "version": 2,
  "conditions": {
    "filterType": "group",
    "logicalOperator": "AND",
    "conditions": [{
      "filterType": "condition",
      "column": "status",
      "operator": "=",
      "value": "complete",
      "logicalOperator": "AND"
    }]
  }
}

When an end signal is configured, the target is evaluated when the signal span arrives OR when max age is reached, whichever occurs first.

Span selectors (trace scope)

For trace-scope jobs, scorers may need only a subset of the spans in a trace rather than the entire trace. Span Selectors let you define named sub-queries that filter, pick, and limit spans within the trace for each scorer.

A span selector defines:

Bind each scorer to a span selector via span selector bindings — a mapping from scorer ID to selector ID. Every scorer in a trace-scope job must have a binding before the job can be activated.

Manual evaluation

You can trigger an evaluation for a specific target on demand, bypassing the automatic sampling and completion logic. This is useful for re-evaluating a trace after changing scorers, or testing a job against a known trace or session.

Send a POST to /api/{org_id}/eval_jobs/{job_id}/manual_eval with:

{
  "targetId": "trace-abc123",
  "traceId": "trace-abc123",
  "sessionId": "session-xyz",
  "variables": { "input": "custom value" },
  "reason": "operator retry after scorer update"
}

The targetId is required. Use traceId or sessionId to pin the evaluation to a specific trace or session. Optional variables override template variables for this evaluation run. The response reports the number of durable evaluation tasks created.

Job lifecycle

Jobs follow a defined state machine:

draft → active ⇄ paused
          ↓
       degraded → active
          ↓
       archived

Use the action buttons on the job detail page to manage lifecycle transitions.

Update a job

Edit any field on a draft or active job. Updating bumps the job's version. If the job is active, the underlying configuration is automatically reconciled:

• Span-scope jobs: the hidden pipeline is updated with new filters, sampling, and scorers.
• Trace/session jobs: if switching TO a span scope, a pipeline is created; if switching FROM a span scope, the old pipeline is torn down.

Scoring pipeline

Span-scope jobs create a PipelineKind::Evaluation pipeline behind the scenes. This pipeline is:

• Hidden from the main Pipeline UI — it is managed exclusively by the eval jobs subsystem.
• Coexisting with user pipelines on the same stream (no "one pipeline per stream" conflict).
• Automatically reconciled when the job is updated.
• Terminating at an LLM evaluation task publisher rather than writing to _llm_scores directly. Durable evaluation tasks are enqueued and processed asynchronously.

Trace-scope and session-scope jobs do NOT create hidden pipelines. Instead, the Eval Scheduler polls trace streams periodically, detects completed targets using the configured idle window and end signal, and publishes evaluation tasks.

Evaluated scores are written to the _llm_scores system stream as LlmScoreRecord entries, and evaluator telemetry (latency, tokens, status) is recorded as OTLP spans in the _evaluator traces stream.

Quality Dashboard

The Quality tab provides a real-time overview of evaluation health across all your score configs, agents, and streams.

Scope filtering

When you drill into a specific score config from the quality page, the detail drawer includes a scope selector that lets you filter KPI cards, trend charts, and the evaluation runs table by target scope: All, Span, Trace, or Session. Switching the scope re-runs all queries within the drawer so you see metrics scoped to the selected granularity.

Evaluation runs

The score config detail drawer includes an Evaluation Runs table that lists individual score records — each row shows the score value (numeric, categorical, or boolean), health classification (healthy/unhealthy), the target identity (scope, trace ID, session ID), agent name, and reasoning if available. Click any row to navigate to the evaluator trace in the _evaluator stream for deeper debugging.

The runs table supports pagination and filtering (all runs or unhealthy only). Scope selector drilling works with the runs table — changing scope narrows the listed runs to only the selected target granularity.

RBAC

Online Evaluations resources have their own OFGA permissions:

Assign the appropriate roles in Identity & Access Management > Roles to control access to evaluation resources.

API Reference

All endpoints are prefixed with /api/{org_id}.

Providers

Score Configs

Scorers

Eval Jobs

Create / Update job payload fields (in addition to fields described above):

Super Cluster

In multi-node deployments, evaluation resources are synchronized across the super cluster via dedicated queue topics (eval_provider, eval_score_config, eval_scorer, eval_job). Changes made on any node propagate automatically.