OpenObserve Docs
User GuideData ExplorationTraces

Traces in OpenObserve

This document explains how to use OpenObserve to collect, view, and analyze distributed traces. It covers how to configure tracing for self-monitoring and external applications, correlate traces with logs, and explore trace data in the UI to identify performance bottlenecks and errors.

Who can use it

All OpenObserve users can view traces.

Where to find it

Select Traces from the left navigation in the OpenObserve UI. Traces

Why traces matter

When an application serves a request, such as placing an order, it often calls many different services behind the scenes. If something slows down or fails, it can be difficult to know where the problem happened. Traces solve this by recording the journey of a single request as it flows through your system.

What is a trace?

A trace is the complete story of one request. It begins when the request enters your system and ends when the final response is sent back. A trace is made up of smaller pieces called spans. Each span represents one operation in the request journey. Examples of operations include verifying a user’s login, reserving inventory, or charging a payment card.

  • Every trace has a trace_id, which is the unique identifier that ties all its spans together.
  • Every span has its own span_id, which shows its place in the request journey.
  • Parent and child relationships between spans make it clear which operation triggered which.

How OpenObserve helps

OpenObserve collects and stores spans from your services. It then reconstructs the request flow in the Traces UI, where you can see how the request moved across different services, how long each step took, and where issues may have occurred.

Example: Retail Application
Imagine a customer clicks place order in your online shop. That single action triggers work across several services:

Browser
├─ API Gateway
    |
    orders-service  		    • operation: POST /checkout
    ├─ auth-service           	• operation: verify token
    ├─ inventory-service      	• operation: reserve items
    ├─ payments-service       	• operation: charge card
    │   └─ payments database  	• operation: SQL query
    └─ email-service          	• operation: send receipt

How this appears as a trace

Root span: orders-service POST /checkout starts when the API receives the request and ends when the response is sent.

Child spans:

  • auth-service verify token.
  • inventory-service reserve items.
  • payments-service charge card.
  • payments database SQL query.
  • email-service send receipt.

Note

  • All spans share one trace_id. Each span has a unique span_id. The parent and child links show who called whom and in what order.
  • The application component that performs an operation is called a service. For example, auth-service, inventory-service, and payments-service are all services involved in processing the order.
  • The specific action performed by the service is called an operation. For example, the auth-service runs verify token, the inventory-service runs reserve items, and the payments-service runs charge card.

Views available in the UI

Trace list: Displays recent traces for the selected time range and selected stream.

Each row shows:

  • Separate Service and Operation name columns. The first span determines these values.
  • The number of spans
  • The request Timestamp
  • The total Duration
  • Badges for the services involved

The Timestamp and Duration column headers are clickable to sort the list in ascending or descending order. The list header shows a badge with the total count of traces found and a separate Error Traces count badge. A pagination control lets you choose the number of rows per page (10, 25, 50, or 100) and navigate between pages. trace-list

Customize columns: Toggle a field's visibility icon in the field sidebar to add or remove it as a table column, drag column headers to reorder them, and click the X on a column header to remove it. Column choices are remembered separately for Traces and Spans modes. Hover any cell to copy its value or add it as an include/exclude filter.

In the field sidebar, fields are organized into groups: Key Fields first, then type groups (String, Number, Boolean), semantic/prefix groups (for example, Kubernetes and HTTP), and Other last. For a single stream, the groups are expanded by default.

Timeline: Shows spans as color-coded horizontal bars. Parent spans contain child spans. Parent time includes the time of child spans, which may run in parallel. Timeline Service map: Shows all services involved in the trace and how long each took. Service map Span details: Provide metadata such as file path, code line, service version, thread ID, and additional attributes. Events and error messages appear when available. Span details


The following configuration steps show how to set up OpenObserve for self-monitoring and for collecting traces from external applications.

Configure for self-monitoring

Steps

Step 1: Navigate to Data Sources and collect authorization details

Go to Data sources > Custom > Traces. opentelemetry-collector-for-traces Note the following values:

  • HTTP Endpoint under OTLP HTTP
  • Authorization header value

Navigate to traces

Step 2: Configure the environment variables

Set the following environment variables on the OpenObserve deployment:

ZO_TRACING_ENABLED=true
ZO_TRACING_SEARCH_ENABLED=true
OTEL_OTLP_HTTP_ENDPOINT=https://<host>/api/<organization>/v1/traces
ZO_TRACING_HEADER_KEY=Authorization
ZO_TRACING_HEADER_VALUE=<paste the Authorization value from the UI>

Replace <host> and <organization> with the values you saw in the UI. Keep /v1/traces. This indicates that from the instance

Step 4: Apply changes

Apply the environment variable changes and restart the service.

Verify and view your first trace
  1. From the left navigation menu, select Logs.
  2. Select a stream and time range, and run a simple log search query.
  3. From the left navigation menu, go to Traces.
  4. Select the trace stream.
  5. Set the time range to the last few minutes, and open the newest trace.
  6. You should see spans from services such as querier, ingester, or scheduler.

Configure to monitor an external application

You can send traces directly from an external application to OpenObserve.

Step 1: Navigate to Data Sources and collect authorization details

In your OpenObserve application, go to Data sources > Custom > Traces. opentelemetry-collector-for-traces From the OTLP HTTP section, copy the following data:

  • HTTP Endpoint
  • Authorization header value

From the OTLP gRPC section, copy the following data:

  • Endpoint
  • Headers: Authorization, organization, stream-name
  • TLS setting insecure or secure
Step 2: Choose how you want to send the data

opentelemetry-collector-for-traces

You can send the data directly from the application using an OpenTelemetry SDK

In this method, add the copied authorization values in the exporter setup within your code:

  • Set the endpoint to the HTTP Endpoint you copied. It ends with /v1/traces.
  • Add the headers exactly as shown in the UI.
  • For HTTP, add Authorization.
  • For gRPC, add Authorization, organization, and stream-name.
  • Set TLS to secure or insecure to match the UI if you use gRPC.

You can also send data through the OpenTelemetry Collector.
To configure this:

  • Open your team's Collector config file.
  • In the exporters section, paste the exporter block exactly as shown in Data sources > Custom > Traces > OTEL Collector for the protocol you use.
Verify and view your first trace
  1. In the external application, perform an operation.
  2. In your OpenObserve instance, go to Traces.
  3. Select the trace stream.
  4. Set the time range to the last few minutes, and open the newest trace.
  5. You should see spans from services such as querier, ingester, or scheduler.

Configure and view log and trace correlation

Configure correlation to navigate between traces and their related log records using shared trace_id and span_id fields.

Understanding log and trace correlation

  • Logs and Traces Correlation: Logs become easy to join with traces when each log record includes the active TraceId and SpanId. The OpenTelemetry logs spec calls this out so that a UI can jump from a span to its related logs and back. Field names can be trace_id and span_id or equivalents that you choose.
  • Trace context: A small set of identifiers that travels with each request so the tracing systems can follow the same request across services. In OpenTelemetry this context is carried in a SpanContext that contains a TraceId and a SpanId and flags.
  • The traceparent header: An HTTP header that passes the current trace context between services. It has four parts in this order: version, trace-id, parent-id, and trace-flags. Example: traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

Requirements

  • Logs and traces must come from the same application run.
  • Your application must write trace context fields into every log record.
  • OpenObserve must know the exact field names you used in your logs.

What makes logs correlatable

The spans are generated by the system or application that you are monitoring. They are created by its tracing library or agent, not by OpenObserve.
A span is active while your code is handling an operation. When a span is active, the logging component can read the following identifiers from it and add them to each log line.

  • trace_id
  • span_id
Step 1: Add trace identifiers to your logs in the application
  1. Ensure your tracing library starts a span for each request and keeps the span active during request processing. Many libraries do this automatically once enabled.
  2. When you write a log record during that time, include the current span identifiers. Add trace_id and span_id to the log record.
  3. Choose log field names that fit your logging format. Examples include trace_id and span_id or traceId and spanId. Make a note of the exact names you used. You will map these field names in OpenObserve in the next step.
Step 2: Tell OpenObserve how your log fields are named

config-custom-trace-id

  1. Open the OpenObserve UI.
  2. Go to Management > Organization Parameters.
  3. Under Log details:
  • Set Trace ID field name to match your log field name. Example trace_id or traceId.
  • Set Span ID field name to match your log field name. Example span_id or spanId.
  1. Save the settings.

Why this matters

Traces in OpenObserve always use trace_id and span_id as defined by OTLP.

Your logs can use different names. The mapping you set here allows the UI to build the correct filter when you open related logs from a span.

Step 3. View the correlation in the UI
  1. Open Traces.
  2. Select the trace stream.
  3. Choose the time range and select Run query.
  4. Open a trace to view its timeline.
  5. Now choose any of these ways to reach the related logs: traces-to-logs A. Use the header stream selector:
  • In the trace header, open the stream dropdown next to View Logs.
  • Pick the log stream that holds your application logs.
  • Select View Logs. The Logs page opens, filtered to the current trace_id and span_id.

B. Use the magnifier on a span row:

  • Hover any span in Operation Name.
  • Select the magnifier icon.
  • It goes to the correlated logs stream and shows the log record which is linked to the trace ID and span ID.

C. Use the span details panel:

  • Select a span to open the detailed span view
  • Select View Logs next to the Span Id.
  • It goes to the correlated logs stream and shows the log record which is linked to the trace ID and span ID.
  1. Open Logs.
  2. Select the appropriate log stream.
  3. Select the time range and select Run query.
  4. Expand a log record.
  5. Select View Trace. view-trace-from-logs
  6. The system shows the trace that matches the trace_id.

Note: If no results appear after selecting View Trace, confirm that in your log stream the trace ID and span ID are available in the trace_id and span_id respectively. If not, ensure the custom trace ID and span ID fields are configured in the Organization parameter section under Management. config-traces-in-org-parameter

View a trace

  1. Go to Traces.
  2. Select the trace stream.
  3. Select time range.
  4. Click Run query.
  5. From the query result, select the desired trace.

Use the Spans | Traces | Service Graph | Service Catalog toggle in the search bar to switch the result list between individual spans and aggregated traces (the default view is Spans), with Service Graph and Service Catalog available on Enterprise. In Spans mode, the result list shows a Spans Found count, and selecting a span row opens its trace with that span focused. Service Catalog shows a per-service metrics table (requests, error rate, latency percentiles, and health status); clicking a service opens Traces filtered by service_name. See Service Catalog for details.

Flame Graph

A Flame Graph view is also available for inspecting a trace. Selecting a span in the flame graph opens that span's details in a resizable bottom panel.

Use Trace Timeline and Service Map

Use Trace Timeline and Service Map to inspect performance and find slow spans: trace-timeline

  1. Go to Traces.
  2. From the stream selector, select the trace stream.
  3. Set a time range.
  4. Select Run query.
  5. From the list of traces, open a trace.
  6. In the trace view, select the Timeline toggle at the top right corner.
  7. In Operation Name, expand the tree and look for the longest bars. The longest bar indicates the most time consuming span. The duration is shown at the right edge of each bar.
  8. Switch to Service Map toggle next to Timeline at the top right corner.
  9. The nodes show which services participated and their time contribution for this trace.
  1. Go to Traces.
  2. From the stream selector, select the trace stream.
  3. Add a filter in the non-SQL editor to find spans with errors, for example, span_status = "ERROR".4. Set a time range.
  4. Select Run query.
  5. From the list of traces, open a trace. Traces should have an error badge. error-badge-spans
  6. In the span tree, click the span highlighted in red warning sign. warning-spans
  7. The detailed span view opens, showing the complete context for that span.
    span-view
  8. If log and trace correlation is configured, select View Logs in the span details panel to open the related logs for further investigation.

Filter by duration

Click the duration field in the field sidebar to view its duration statistics: P25, P50, P75, P95, P99, and Max (the maximum observed duration). Each statistic provides one-click buttons to add a duration >= <value> or duration <= <value> filter to the query.

Duration filters also accept human-readable unit suffixes, for example:

  • duration >= '1.50ms'
  • duration >= '2 seconds'
  • duration <= '100us'

The accepted units are us (or µs), ms, s, and m. OpenObserve auto-converts these values to microseconds.

Enterprise

While a query runs, a Cancel button replaces Run query, allowing you to abort in-flight queries.

Explore the detailed span view

detailed-span-view

TabWhat it showsWhen to use
TagsKey/value attributes on the span (code location, service identity, status, parent links, etc.).Identify where the work ran, whether it showed an error, and the parent/child relationship.
ProcessCompact identity of the emitting process (service, instance, version).Confirm the exact instance/version for rollbacks or targeted restarts.
EventsTime-ordered records emitted inside the span (INFO/ERROR messages with context).Read the span's timeline narrative and error messages without leaving the trace.
AttributesThe full JSON document is stored for this span.Copy/paste into tickets, verify raw values, or see fields not surfaced elsewhere.
ErrorConsolidated error indicators for the span: HTTP status, gRPC status/name/message, exception events, error type, DB response status, and process exit code.Quickly assess what failed and why without scanning across multiple tabs.
DBDatabase call details: DB system, the query/statement (syntax-highlighted by db_system), collection/table, rows returned, batch size, response status code, and a query summary. Appears only when the span has db_* attributes.Inspect the database operation a span performed.
MetricsCorrelated metric streams for the span's service. Use the selection pills (Essentials, Compute, Memory, Storage, Network, All) together with the Pod / Node scope chips to choose which metrics to display. Essentials is the default when curated streams exist.Inspect the resource health of the service that emitted the span without leaving the trace. When no matching metrics exist, the view reads No Data Found.

Enterprise

For LLM traces that have an associated LLM-evaluation pipeline producing evaluation records, the trace detail view shows an additional Evaluations tab. It displays the evaluation results for the trace. This tab appears only when such evaluation records exist for the trace.

Was this page helpful?

Last updated on

Traces Overview

Traces in OpenObserve show how requests flow across services in distributed systems, helping you pinpoint performance bottlenecks and troubleshoot errors.

Service Graph

Service Graph visualizes how services communicate as a real-time topology from distributed traces, with health colors, call edges, and inferred dependencies.

On this page

Why traces matterWhat is a trace?How OpenObserve helpsHow this appears as a traceViews available in the UIConfigure for self-monitoringConfigure to monitor an external applicationConfigure and view log and trace correlationUnderstanding log and trace correlationRequirementsWhat makes logs correlatableView a traceUse Trace Timeline and Service MapUse filters to find the related logsFilter by durationExplore the detailed span view