OpenObserve Docs
IntegrationAITools

Promptfoo → OpenObserve

Updated View MarkdownOpen in ChatGPTOpen in Claude

Capture a trace for every Promptfoo test case, with the model call, token usage, and pass/fail result. Promptfoo is an open-source CLI for evaluating and red-teaming LLM prompts and apps. It has built-in OpenTelemetry tracing: turn it on in promptfooconfig.yaml and point its OTLP exporter at OpenObserve.

Prerequisites

  • Node.js 22.22+
  • An OpenObserve account (cloud or self-hosted)
  • Your OpenObserve organisation ID and Base64-encoded auth token
  • An OpenAI API key

Installation

Run Promptfoo with npx, or install it globally:

npm install -g promptfoo

Configuration

Set these environment variables in the shell that runs the eval:

export OPENAI_API_KEY=your-openai-api-key
export PROMPTFOO_OTEL_ENDPOINT=https://api.openobserve.ai/api/your_org_id/v1/traces
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic%20<your_base64_token>"
export PROMPTFOO_OTEL_SERVICE_NAME=promptfoo
VariableDescription
PROMPTFOO_OTEL_ENDPOINTFull OTLP/HTTP traces URL, including /v1/traces. OTEL_EXPORTER_OTLP_ENDPOINT is used when this is not set.
OTEL_EXPORTER_OTLP_HEADERSHeaders for the exporter. Encode the space after Basic as %20.
PROMPTFOO_OTEL_SERVICE_NAMEservice_name on the spans. Default promptfoo.

Instrumentation

Add tracing.enabled: true to promptfooconfig.yaml. No code changes are needed: Promptfoo creates the spans for its built-in providers, including OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Google Vertex AI, Ollama, and the HTTP provider.

description: Tracing demo

tracing:
  enabled: true

prompts:
  - 'Answer in one sentence: {{question}}'

providers:
  - openai:gpt-4o-mini

tests:
  - vars:
      question: What is distributed tracing?
    assert:
      - type: icontains
        value: trace
  - vars:
      question: What is an OpenTelemetry span?
    assert:
      - type: icontains
        value: span
      - type: llm-rubric
        value: Explains that a span is one unit of work in a trace

Run the eval:

npx promptfoo@latest eval

Promptfoo also keeps the traces in its local database, so they still show in its own web viewer. Set PROMPTFOO_OTEL_LOCAL_EXPORT=false to send them only to OpenObserve.

What Gets Captured

Each test case produces one trace. OpenObserve stores the dotted attribute names with underscores.

promptfoo.test_case
├── openai:gpt-4o-mini              (target provider call)
│   └── chat gpt-4o-mini
├── grader icontains                (one per assertion)
└── grader llm-rubric
    └── grader provider <grader>    (model-graded assertions only)

promptfoo.test_case span (root)

AttributeDescription
promptfoo_test_successtrue if every assertion passed
promptfoo_test_scoreTest case score
promptfoo_eval_idEval run ID
promptfoo_test_indexTest case index in the eval
promptfoo_provider_idProvider under test

chat <model> span (model call)

AttributeDescription
gen_ai_provider_nameProvider (e.g. openai)
gen_ai_operation_nameOperation (e.g. chat)
gen_ai_request_modelModel requested
gen_ai_response_modelModel that served the response
gen_ai_usage_input_tokensInput tokens
gen_ai_usage_output_tokensOutput tokens
gen_ai_response_finish_reasonsFinish reasons
promptfoo_provider_idPromptfoo provider ID (e.g. openai:gpt-4o-mini)

grader <assertion> span

AttributeDescription
gen_ai_evaluation_nameAssertion type (e.g. icontains, llm-rubric)
gen_ai_evaluation_score_labelpass or fail
gen_ai_evaluation_score_valueGrader score
gen_ai_evaluation_explanationGrader reason

The target provider span (openai:gpt-4o-mini) carries promptfoo_cache_hit. Cached responses do not call the model, so run with --no-cache to trace every call.

Viewing Traces

  1. Log in to OpenObserve and navigate to Traces
  2. Filter by service_name = promptfoo to isolate eval traces
  3. Filter by operation_name = promptfoo.test_case to list test cases
  4. Filter by promptfoo_test_success = false to find failing cases
  5. Open a trace to see the model call and grader spans under the test case

Promptfoo trace in OpenObserve, Waterfall view of a promptfoo.test_case with the model call and grader spans and the grader llm-rubric attributes

Next Steps

With Promptfoo traces in OpenObserve, you can track pass rates and token usage across eval runs, alert when promptfoo_test_success drops, and compare models by latency.

Read More

Was this page helpful?

On this page