LLM Playground

The Playground is an interactive workbench for iterating on prompts, models, and scorers before you commit them to a Dataset or an Experiment. Run a model to stream its answer, score the result to read your judge's reasoning, and share a snapshot to capture the whole board.

Overview

The Playground is a two-dimensional board:

• Columns are variants: each column is one prompt configuration, a provider, an optional model, an ordered list of messages, sampling parameters, and optional tools or a response format.
• Rows are cases: each row is an input (plus optional expected output and metadata) that every variant runs against.

A cell is the intersection of a variant and a case. Run a cell to stream the model's answer back live; score a cell to run your scorers against it and read each judge's verdict and reasoning.

The Playground doesn't save to the server as you work, so drafts never affect your evaluation analytics. Your recent drafts are kept in your browser only (see Recent Drafts below). Share is the only action that saves to the server: it copies every column, row, result, and score into a snapshot, which keeps rendering even after the underlying dataset, scorer, or provider changes or is deleted.

Open the Playground

Navigate to AI > Experiment > Playground, or start from something you already have:

• From a prompt: on a prompt in Prompts, click Open in Playground. The column loads that prompt version, and a chip such as support-answer@v3 shows where it came from.
• From a trace: open an LLM span in trace details and click Open in Playground. The span's messages load into a column marked From trace span.

The toolbar holds the board-wide actions: Recent Drafts (the last 10 boards in this browser), Reset, a counter of variant windows (up to 4), Score, Share, and Run All.

Build a variant

Each column is a variant. Configure it with:

Variables

Templates in message content bind variables from the row's input:

• A string input binds {{input}}.
• An object input binds each of its fields by name.

Type {{ in a message to get a list of the variables you can use. The Variables menu shows every variable the board uses, lets you set test values, and lets you add new ones. {{expected_output}} is never available in task prompts; reference answers only feed scorers.

Only string content is templated. Structured content (multimodal parts and tool results) is passed through as-is, so its structure is never corrupted.

Add cases

Each row is a case. A case supplies the input every variant renders against, and optionally:

Run a cell

Running a cell streams the model's answer back live. You first see the exact rendered prompt (with variables filled in), then incremental text as it's generated, and finally the model used, latency, and token usage and cost.

Configuration errors (an inaccessible provider, an unrenderable message, an invalid parameter) are caught before streaming starts. Once streaming begins, an upstream failure, including rate limits and other gateway errors, is surfaced as an error in the stream. Any API key in an upstream error is redacted before it reaches you.

Score a cell

Scoring a cell runs your scorers against it and returns one verdict per scorer immediately; nothing is saved, so throwaway scores never affect your experiment comparisons. Select the scorers to apply, then run them.

Each result reports one of three outcomes:

Scorers run at their latest version in the Playground. Pinning to a specific version happens when you promote a column to an Experiment.

Tool calls and structured output

When a variant defines tools, a model that chooses to call one returns the call (name and arguments) as the cell's output rather than executing anything. This lets you iterate on tool definitions without wiring up an execution environment.

When a variant sets a response format, the provider is asked for structured output, and the returned structure is shown as the result. Open Structured Output on a column and turn on Constrain output to a JSON schema to set one.

Not every provider supports JSON schemas, so the column badge tells you what happens to yours:

Share a snapshot

Share captures the entire board, every column, row, result, and score, into an immutable snapshot and returns a short link you can hand to a teammate. The snapshot keeps rendering even after the dataset, scorer, or provider it came from changes or is deleted.

Editing a shared board doesn't update the existing snapshot; it creates a new one. A snapshot is visible only within your organization. To build on a snapshot someone shared with you, open it and click Continue From This. You then work on a copy, and the original stays unchanged.

Limits and retention

• A snapshot may hold up to 4 columns, 10 rows, and 1 MB of data.
• Snapshots expire after a configurable number of days of not being opened; opening one renews the window.
• Each organization is capped at a configurable number of snapshots; over the cap, the least recently opened are purged first.

Save a prompt or create an experiment

The Playground is a drafting surface for the rest of the LLM evaluations suite. When you're happy with a column:

• Click Save as Prompt to store it in Prompts. Choose Create a new prompt, or Append to an existing prompt to save it as that prompt's next version.
• Click Create Experiment to turn the column into an Experiment. This is the moment each scorer is pinned to a specific version.

To use cases from a Dataset, click Sample from Dataset. The sampled rows fill your variables, such as {{input}} and {{expected_output}}, so you can run the board against real cases.

Configuration

The Playground is part of AI Observability in the Enterprise edition and OpenObserve Cloud.

RBAC

The Playground has its own permissions (GET, LIST, POST, PUT, DELETE). Two things to note:

• Run and Score require the write grant, even though they save nothing, since both call a provider or judge and spend real money.
• You can only use providers and scorers you have access to.

Assign the appropriate roles in IAM > Roles to control Playground access.

API Reference

All endpoints are prefixed with /api/{org_id}.

Run payload (a column plus an optional row):

{
  "column": {
    "provider_id": "provider-id",
    "model": "gpt-4o",
    "messages": [
      { "role": "system", "content": "You are a helpful assistant." },
      { "role": "user", "content": "{{input}}" }
    ],
    "params": { "temperature": 0.2, "max_tokens": 256 },
    "tools": [],
    "response_format": null
  },
  "row": {
    "input": "What is the retention default?",
    "expected_output": "14 days",
    "metadata": {}
  }
}

Score payload (the scorers to run against a subject):

{
  "scorer_ids": ["scorer-1", "scorer-2"],
  "input": "What is the retention default?",
  "output": "Fourteen days.",
  "expected_output": "14 days",
  "metadata": {}
}

Share payload (the whole workbench by value):

{
  "payload": {
    "columns": [ { "provider_id": "provider-id", "messages": [], "model": "gpt-4o", "params": {} } ],
    "rows": [ { "input": "case 1" } ]
  },
  "parent_snapshot_id": null
}