LLM Playground

The Playground is an interactive workbench for iterating on prompts, models, and scorers before you commit them to a Dataset or an Experiment. Run a model to stream its answer, score the result to read your judge's reasoning, and share a snapshot to capture the whole board.

Overview

The Playground is a two-dimensional board:

• Columns are variants: each column is one prompt configuration, a provider, an optional model, an ordered list of messages, sampling parameters, and optional tools or a response format.
• Rows are cases: each row is an input (plus optional expected output and metadata) that every variant runs against.

A cell is the intersection of a variant and a case. Run a cell to stream the model's answer back live; score a cell to run your scorers against it and read each judge's verdict and reasoning.

The Playground is volatile by design: opening or running it saves nothing, so drafts never affect your evaluation analytics. Share is the only action that saves anything: it copies every column, row, result, and score into a snapshot, which keeps rendering even after the underlying dataset, scorer, or provider changes or is deleted.

Build a variant

Each column is a variant. Configure it with:

Variables

Templates in message content bind variables from the row's input:

• A string input binds {{input}}.
• An object input binds each of its fields by name.

Only string content is templated. Structured content (multimodal parts and tool results) is passed through as-is, so its structure is never corrupted.

Add cases

Each row is a case. A case supplies the input every variant renders against, and optionally:

Run a cell

Running a cell streams the model's answer back live. You first see the exact rendered prompt (with variables filled in), then incremental text as it's generated, and finally the model used, latency, and token usage and cost.

Configuration errors (an inaccessible provider, an unrenderable message, an invalid parameter) are caught before streaming starts. Once streaming begins, an upstream failure, including rate limits and other gateway errors, is surfaced as an error in the stream. Any API key in an upstream error is redacted before it reaches you.

Score a cell

Scoring a cell runs your scorers against it and returns one verdict per scorer immediately; nothing is saved, so throwaway scores never affect your experiment comparisons. Select the scorers to apply, then run them.

Each result reports one of three outcomes:

Scorers run at their latest version in the Playground. Pinning to a specific version happens when you promote a column to an Experiment.

Tool calls and structured output

When a variant defines tools, a model that chooses to call one returns the call (name and arguments) as the cell's output rather than executing anything. This lets you iterate on tool definitions without wiring up an execution environment.

When a variant sets a response format, the provider is asked for structured output, and the returned structure is shown as the result.

Share a snapshot

Share captures the entire board, every column, row, result, and score, into an immutable snapshot and returns a short link you can hand to a teammate. The snapshot keeps rendering even after the dataset, scorer, or provider it came from changes or is deleted.

Editing a shared board doesn't update the existing snapshot; it creates a new one. A snapshot is visible only within your organization.

Limits and retention

• A snapshot may hold up to 4 columns, 10 rows, and 1 MB of data.
• Snapshots expire after a configurable number of days of not being opened; opening one renews the window.
• Each organization is capped at a configurable number of snapshots; over the cap, the least recently opened are purged first.

Promote to a Dataset or Experiment

The Playground is a drafting surface for the rest of the LLM evaluations suite. When you're happy with a result, promote it:

• Promote a result cell into a Dataset, where it becomes a reference case recorded with source playground.
• Promote a column into an Experiment, which is the moment a scorer is pinned to a specific version.

Configuration

The Playground is part of the enterprise AI Observability / Online Evaluations suite.

RBAC

The Playground has its own permissions (GET, LIST, POST, PUT, DELETE). Two things to note:

• Run and Score require the write grant, even though they save nothing, since both call a provider or judge and spend real money.
• You can only use providers and scorers you have access to.

Assign the appropriate roles in Identity & Access Management > Roles to control Playground access.

API Reference

All endpoints are prefixed with /api/{org_id}.

Run payload (a column plus an optional row):

{
  "column": {
    "provider_id": "provider-id",
    "model": "gpt-4o",
    "messages": [
      { "role": "system", "content": "You are a helpful assistant." },
      { "role": "user", "content": "{{input}}" }
    ],
    "params": { "temperature": 0.2, "max_tokens": 256 },
    "tools": [],
    "response_format": null
  },
  "row": {
    "input": "What is the retention default?",
    "expected_output": "14 days",
    "metadata": {}
  }
}

Score payload (the scorers to run against a subject):

{
  "scorer_ids": ["scorer-1", "scorer-2"],
  "input": "What is the retention default?",
  "output": "Fourteen days.",
  "expected_output": "14 days",
  "metadata": {}
}

Share payload (the whole workbench by value):

{
  "payload": {
    "columns": [ { "provider_id": "provider-id", "messages": [], "model": "gpt-4o", "params": {} } ],
    "rows": [ { "input": "case 1" } ]
  },
  "parent_snapshot_id": null
}