LLM Experiments

Experiments bring offline, batch evaluation to LLM Evaluations. Where Online Evaluations score live traces as they arrive, Experiments run a fixed set of Dataset cases through a task and scorers, then let you compare two runs to see whether a change actually improved your application.

Overview

Enable Experiments

Experiments are part of the LLM Evaluations feature in the Enterprise edition and OpenObserve Cloud. It is on by default:

O2_EVAL_ENABLED=true

This adds AI > Annotate > Datasets and AI > Experiment > Experiments and Remote Tasks to the navigation. You'll also need at least one Provider, Score Config, and Scorer (see LLM Evaluations).

Datasets

Navigate to AI > Annotate > Datasets, click New Dataset, and give it a name, description, and tags. Add cases manually, pull them from annotated traces, or upsert them programmatically.

Cases

A case pairs an input with an optional expected_output (a reference answer). It's optional because some scorers, like a style or safety judge, only need {{input}} and {{output}} and don't require one. Each case records its source: Manual, Trace (promoted from a captured trace), Annotation (from an annotation queue), or Playground.

Every write to a Dataset creates a new version, so you can read it as it existed at any past point. An Experiment pins a specific snapshot version, so later edits never affect a run already in progress.

Programmatic upserts

The Dataset upsert API creates or updates cases by a client-supplied logicalId. Key fields:

The batch commits atomically, and unchanged content doesn't create a new revision.

Experiments

The Experiments list shows each run's status, progress, and cost, filterable by Dataset.

Create an experiment

Navigate to AI > Experiment > Experiments and click New Experiment.

Click Create & run to start the experiment.

Task types

Preview and cost estimate

Before creating, the form previews the plan: row/slot counts, pinned scorer versions, and any cases missing a reference answer a scorer needs. It also estimates cost in USD (slots × tokens, using known model pricing). Plans are capped at 10,000 slots.

If the estimate is above the warning threshold (default $10), Create & run opens a Run this costly experiment? dialog that shows the estimated cost. Click Create Anyway to run it.

Lifecycle

pending → running → completed
              ↓
          cancelled
              ↓
            failed → (retry) → running

An Experiment starts running as soon as it's created, within a 24-hour window. A failed run can be retried without restarting finished slots. You can also retry a single row: the retry is queued, and the row updates when it finishes.

Baseline and cloning

Each Dataset can have one Baseline Experiment: the default comparison target. Only a completed run with no task errors is eligible; setting a new Baseline replaces the old one. You can also clone an Experiment to reuse its definition, with a (copy) suffix.

Experiment results

The detail page shows every slot's status and score, in plan order, including slots still pending, so a live run never hides unfinished work.

Each slot reports a task status (pending, in_progress, ok, skipped, error) and a score per pinned scorer (plus any client-reported scores for SDK Experiments). Run-level summaries show progress, skip reasons, and per-scorer aggregates, plus an overall scoring status (pending/running/completed/completed_with_errors). Comparisons and CI checks should wait for completed.

For SDK Experiments, results come back through an ingest API that accepts records one at a time: a bad record doesn't block the rest, and resending an accepted one is a no-op.

Comparing experiments

On the Experiments list, select two experiments from the same Dataset and click Compare to pit a candidate against a Baseline. Cases are matched by logicalId and bucketed:

A dimension only "votes" (is gating) if its Score Config declares a direction (gte/lte), a healthy category, or a healthy boolean. Cost and latency always favor lower. Everything else is descriptive: visible, but not a verdict. Each delta is also checked against the noise between a case's own trials, so you can tell a real change from a fluke.

Remote Tasks

A Remote Task is an HTTP service that produces the output for each row, such as your RAG pipeline or agent behind an endpoint. You publish it once, and every experiment pins the exact version it ran against. Navigate to AI > Experiment > Remote Tasks and click New Task.

Use the Test Connection panel to send a sample input, from a Dataset or typed in by hand, and see the request and response. Then click Publish, or Save as Draft to come back later. A draft can't be referenced by an experiment until a test connection passes.

After publishing:

• Test Run runs the latest published version against up to ten samples. Nothing is recorded.
• Replace Credentials swaps the stored token or password in place. The task keeps its version, and the stored value is never shown.
• The Signing tab rotates the signing key in four steps: Create Candidate, Test Candidate, Activate Candidate, then let the previous key expire after a grace period of up to 24 hours (or click End Grace Now).
• Deleting a task retires it. Published versions stay for history but can no longer be referenced.

See LLM Evaluations for the related Remote scorer type.

Configuration

RBAC

Datasets, Experiments, and Remote Tasks each have their own permissions (GET, LIST, POST, PUT, DELETE). Assign roles in IAM > Roles.

API Reference

All endpoints are prefixed with /api/{org_id}.

Create-Experiment payload highlights: datasetVersion, datasetFilter (subset filter), task (tagged union: inline_prompt/prompt_ref/remote/sdk), scorers (id + version), trialCount, idempotencyKey, confirmCostEstimate (required above the cost warning threshold).