OpenTelemetry for Erlang: Instrument OTP Apps with OpenObserve

Erlang systems are built for observability in a sense — you can trace function calls on a live node with dbg, inspect any process with observer, and hot-load debug code in production. What OTP does not give you is distributed request tracing that connects your Erlang node to the Go service that called it and the database behind it, in a vendor-neutral format. That is what OpenTelemetry adds. This guide walks through instrumenting an Erlang/OTP application with the OpenTelemetry Erlang SDK — rebar3 dependencies, sys.config, the ?with_span macros — and shipping the data to OpenObserve over OTLP.

If you are new to OpenTelemetry itself — the signals, the collector, OTLP — start with What is OpenTelemetry? A Complete Guide and come back here for the Erlang specifics.

This page is about Erlang proper: rebar3, sys.config, Erlang syntax. If you work in Elixir, the same underlying libraries apply but the setup differs (Mix, config/runtime.exs, the OpenTelemetry.Tracer wrapper) — use the Elixir guide instead.

Why OpenTelemetry for Erlang

The OpenTelemetry Erlang implementation is maintained in the opentelemetry-erlang repository and published to Hex as ordinary OTP applications, so it fits the standard rebar3 + relx workflow:

• Vendor-neutral instrumentation. Instrument once, export anywhere OTLP is accepted — no proprietary tracing library baked into your code.
• Idiomatic OTP design. The SDK is a supervised OTP application; the batch span processor is a gen_statem; configuration happens in sys.config or OS environment variables like everything else on your node. If the SDK crashes, its supervision tree restarts it without touching your application.
• A clean API/SDK split. Library authors depend only on opentelemetry_api (cheap, no-op by default); the final release decides whether to include the opentelemetry SDK and an exporter. Your library users pay nothing unless they opt in.

Signal stability in Erlang

Be precise about what “supported” means. As of mid-2026, the state of the Erlang implementation is:

Signal	Status	Packages	Notes
Traces	Stable	opentelemetry_api 1.5.x, opentelemetry 1.7.x, opentelemetry_exporter 1.10.x	Production-ready; OTLP export over HTTP or gRPC
Metrics	In development	opentelemetry_api_experimental 0.5.x, opentelemetry_experimental 0.5.x	API and SDK usable, but the released packages ship only a console exporter — no OTLP metrics exporter has been released yet
Logs	In development	otel_log_handler in opentelemetry_experimental 0.5.x	An experimental OTP logger handler exists, but OTLP log export is not in a released package

Traces are the headline: the tracing API, SDK, and OTLP exporter are all 1.x, spec-compliant, and safe for production. Metrics and logs are genuinely experimental — the code lives in 0.x packages whose APIs can change, and crucially, the OTLP export path for both signals is not available in any released Hex package as of opentelemetry_exporter 1.10.0 and opentelemetry_experimental 0.5.1. The repository’s main branch carries partial supporting infrastructure (protobuf modules such as opentelemetry_exporter_metrics_service_pb, gRPC service modules, and dispatch code that references an OTLP metrics exporter), but the actual exporter module has not been released. This guide shows what works today and names what doesn’t, with pragmatic workarounds for metrics and logs.

Prerequisites

• Erlang/OTP 23 or later (the SDK targets OTP 23+; use a recent OTP 26/27 in practice) and rebar3.
• An OpenObserve instance. Either an OpenObserve Cloud account or a self-hosted instance. For local testing, a single container is enough:

docker run -d --name openobserve \
  -p 5080:5080 -p 5081:5081 \
  -e ZO_ROOT_USER_EMAIL="root@example.com" \
  -e ZO_ROOT_USER_PASSWORD="Complexpass#123" \
  public.ecr.aws/zinclabs/openobserve:latest

• Your OpenObserve credentials. OTLP ingestion uses HTTP Basic auth. Generate the token once:

echo -n 'root@example.com:Complexpass#123' | base64

This prints cm9vdEBleGFtcGxlLmNvbTpDb21wbGV4cGFzcyMxMjM= — the value you will pass in the Authorization: Basic header.

In OpenObserve Cloud, the ingestion page under Data Sources shows your organization-specific endpoint and a ready-made Basic auth token, so you rarely need to construct it by hand.

Zero-code instrumentation: the short answer for Erlang

There is no BEAM agent. Unlike Java’s -javaagent or Python’s opentelemetry-instrument, nothing attaches to a running Erlang node and instruments it from outside. The Erlang model is explicit and release-driven:

1. Your application code depends on opentelemetry_api and calls its macros. With no SDK present, every call is a cheap no-op.
2. Your release includes the opentelemetry SDK and opentelemetry_exporter, which boot before your application and turn those no-ops into real spans.

What softens the “manual” part is the opentelemetry-erlang-contrib repository, which carries instrumentation libraries for common servers and clients — opentelemetry_cowboy and opentelemetry_elli create a server span per HTTP request, opentelemetry_grpcbox handles gRPC, and more. Much of contrib is Elixir-ecosystem-focused (Phoenix, Ecto, Oban), but the Cowboy, Elli, and grpcbox pieces are plain Erlang. Wire one of those in and you get request-level spans without touching handler code; everything below the request boundary is yours to instrument with the macros in the next section.

Project setup: rebar3, the release, and sys.config

Add the three core packages to rebar.config:

{deps, [opentelemetry_api,
        opentelemetry,
        opentelemetry_exporter]}.

In your application’s .app.src, list only the API — your code compiles and runs against it whether or not an SDK is present:

{applications,
 [kernel,
  stdlib,
  opentelemetry_api]}.

The SDK and exporter belong in the release, ordered so they start before your application. Starting them as temporary is the recommended pattern: if telemetry ever fails permanently, it must never take the node down with it.

{relx, [{release, {checkout, "1.4.2"},
         [{opentelemetry_exporter, temporary},
          {opentelemetry, temporary},
          checkout,
          sasl]},
        {sys_config_src, "config/sys.config.src"},
        {vm_args_src, "config/vm.args.src"}]}.

The SDK starts its supervision tree at boot and reads its configuration from the application environment (or OS environment variables — more on those later), so config/sys.config.src is where the initial wiring lives:

[
 {opentelemetry,
  [{span_processor, batch},
   {traces_exporter, otlp},
   {resource, [{service, #{name => "checkout-service",
                           version => "1.4.2"}},
               {deployment, #{environment => "production"}}]}]},

 {opentelemetry_exporter,
  [{otlp_protocol, http_protobuf},
   {otlp_endpoint, "http://localhost:5080/api/default"},
   {otlp_headers, [{"authorization", "Basic cm9vdEBleGFtcGxlLmNvbTpDb21wbGV4cGFzcyMxMjM="}]}]}
].

Notes on this file:

• span_processor => batch is the production choice; spans are queued and exported in the background by a gen_statem rather than blocking the process that ended the span.
• The nested resource structure is flattened with dots: {service, #{name => ...}} becomes the service.name resource attribute.
• The otlp_endpoint shown targets self-hosted OpenObserve; the exporter appends v1/traces to the path itself. The full OpenObserve story — Cloud values, environment variables, and one important Erlang-specific header gotcha — is covered in its own section below.
• During development you can swap {traces_exporter, otlp} for {traces_exporter, {otel_exporter_stdout, []}} to print spans to the shell and confirm instrumentation before any network is involved.

Manual instrumentation

Traces

Include otel_tracer.hrl and the shared record/status definitions, then use the macros. ?with_span/3 starts a span, runs the fun, and reliably ends the span even if the fun throws:

-module(checkout).

-include_lib("opentelemetry_api/include/otel_tracer.hrl").
-include_lib("opentelemetry_api/include/opentelemetry.hrl").

-export([process_order/1]).

process_order(OrderId) ->
    ?with_span(<<"process-order">>,
               #{attributes => #{'order.id' => OrderId}},
               fun(_SpanCtx) ->
                       Total = calculate_total(OrderId),
                       charge_payment(OrderId, Total)
               end).

calculate_total(OrderId) ->
    %% Child span: automatically parented to process-order because the
    %% parent span is in this process's current context.
    ?with_span(<<"calculate-total">>, #{},
               fun(_SpanCtx) ->
                       Items = cart_db:items(OrderId),
                       ?set_attribute('cart.items', length(Items)),
                       lists:sum([P || {_Sku, P} <- Items])
               end).

charge_payment(OrderId, AmountCents) ->
    ?with_span(<<"charge-payment">>, #{attributes => #{'payment.provider' => <<"stripe">>}},
               fun(_SpanCtx) ->
                       try payment_client:charge(OrderId, AmountCents) of
                           {ok, Ref} ->
                               ?add_event(<<"payment.captured">>, #{'payment.ref' => Ref}),
                               {ok, Ref};
                           {error, Reason} ->
                               ?set_status(?OTEL_STATUS_ERROR, atom_to_binary(Reason)),
                               {error, Reason}
                       catch
                           Class:Reason:Stacktrace ->
                               ?record_exception(Class, Reason, Stacktrace, #{}),
                               ?set_status(?OTEL_STATUS_ERROR, <<"payment crashed">>),
                               erlang:raise(Class, Reason, Stacktrace)
                       end
               end).

The macros that matter day to day: ?with_span/3 (span around a fun), ?start_span/?end_span (when a fun doesn’t fit, e.g. spanning gen_server callbacks), ?set_attribute/?set_attributes, ?add_event, ?record_exception, ?set_status, and ?current_span_ctx. ?set_status(?OTEL_STATUS_ERROR, ...) is what makes failed operations filterable in the OpenObserve Traces UI — an uncaught exception inside ?with_span sets it for you, but handled errors need it explicitly.

Context is per-process. The current span lives in the calling process’s context (implemented on the process dictionary), which has two consequences worth internalizing:

1. Plain function calls in the same process nest spans automatically — that is why calculate-total parents correctly above.
2. A spawned process starts with an empty context. To keep a trace connected across processes, hand the context over explicitly:

Ctx = otel_ctx:get_current(),
proc_lib:spawn_link(fun() ->
    otel_ctx:attach(Ctx),
    ?with_span(<<"send-confirmation-email">>, #{},
               fun(_) -> mailer:send(OrderId) end)
end).

The same rule applies to gen_server:cast and friends: include otel_ctx:get_current() in the message and otel_ctx:attach/1 in the callback if the work should belong to the caller’s trace.

Crossing service boundaries uses the W3C traceparent header via the configured text-map propagator (trace context + baggage by default). Inject on the client side, extract on the server side:

%% Outgoing httpc call: inject traceparent/baggage into the header list.
Headers = otel_propagator_text_map:inject([]),
httpc:request(get, {"http://inventory:8081/stock?sku=42", Headers}, [], []).

%% Incoming request (if not using opentelemetry_cowboy/opentelemetry_elli,
%% which do this for you): extract before starting the server span.
otel_propagator_text_map:extract(ReqHeaders).

Metrics

Metrics in Erlang are experimental and live in two additional packages. Add them to rebar.config and the release if you want to try them:

{deps, [opentelemetry_api,
        opentelemetry,
        opentelemetry_exporter,
        opentelemetry_api_experimental,
        opentelemetry_experimental]}.

The API mirrors the tracing macros, from otel_meter.hrl:

-module(checkout_metrics).

-include_lib("opentelemetry_api_experimental/include/otel_meter.hrl").

-export([init/0, order_processed/1]).

init() ->
    ?create_counter('orders.processed',
                    #{description => <<"Number of orders processed">>,
                      unit => '{order}'}),
    ?create_histogram('payment.duration',
                      #{description => <<"Payment processing duration">>,
                        unit => ms}).

order_processed(DurationMs) ->
    ?counter_add('orders.processed', 1, #{'payment.provider' => <<"stripe">>}),
    ?histogram_record('payment.duration', DurationMs, #{}).

A periodic reader collects and exports on an interval, configured under opentelemetry_experimental in sys.config:

{opentelemetry_experimental,
 [{readers, [#{module => otel_metric_reader,
               config => #{export_interval_ms => 30000,
                           exporter => {otel_metric_exporter_console, #{}}}}]}]}

Here is the honest status, and it matters: the released packages cannot ship metrics over OTLP yet. As of opentelemetry_experimental 0.5.1 and opentelemetry_exporter 1.10.0, the only exporter you can actually configure from Hex releases is the console exporter shown above; no OTLP metrics exporter module has been released. The main branch contains partial infrastructure (protobuf and gRPC service modules plus dispatch code referencing an OTLP metrics exporter), but the exporter module itself is not in any released package. So treat Erlang metrics as something to experiment with locally, not a production pipeline. Until an OTLP metrics release lands, teams that need metrics in OpenObserve today typically export them outside the OTel SDK — for example exposing a Prometheus endpoint with an Erlang Prometheus client and scraping it with an OpenTelemetry Collector (prometheus receiver → otlphttp exporter → OpenObserve), or deriving RED metrics from the traces you are already sending.

Logs

The logs signal is also in development. What exists in released packages is otel_log_handler in opentelemetry_experimental — an OTP logger handler that batches log events — but, as with metrics, no OTLP log exporter has shipped, so there is no released end-to-end path from logger to an OTLP backend.

What works today, and works well, is the classic BEAM pattern with one OpenTelemetry-flavored upgrade: the Erlang SDK automatically stamps trace context into logger process metadata. Whenever a span becomes current in a process, the API calls logger:update_process_metadata/1 with otel_trace_id, otel_span_id, and otel_trace_flags. Emit structured logs including that metadata, ship them to OpenObserve, and you can pivot between a trace and its exact log lines even though the logs never touch the OTel SDK’s log pipeline:

%% Anywhere inside a ?with_span fun — otel_trace_id/otel_span_id are already
%% in this process's logger metadata:
?LOG_INFO("order processed", #{order_id => OrderId, amount_cents => Total}).

Configure a JSON template that includes the OTel fields — for example with the built-in logger_formatter or a JSON formatter such as logger_formatter_json, adding otel_trace_id and otel_span_id to the template — and write to stdout. From there, two solid routes into OpenObserve:

1. OpenTelemetry Collector: filelog receiver (or journald/container log collection) → otlphttp exporter pointed at your OpenObserve endpoint.
2. Direct HTTP ingestion: ship the JSON lines to OpenObserve’s /api/<org>/<stream>/_json endpoint with FluentBit or Vector.

Either way, keep the field names otel_trace_id/otel_span_id (or rename them to trace_id/span_id in your formatter) so trace-to-log correlation queries are trivial.

Sending data to OpenObserve

OpenObserve ingests OTLP natively — no collector required in between, although running one is a fine architecture too. Two transports are supported:

• OTLP/HTTP at <host>/api/<org>/v1/traces (port 5080 self-hosted). This is the exporter’s default protocol (http_protobuf).
• OTLP/gRPC on port 5081 (self-hosted), which additionally requires an organization header alongside Authorization. The gRPC transport uses grpcbox, which is fetched automatically as a transitive dependency of opentelemetry_exporter, so no separate dependency entry is required.

Stick with HTTP unless you have a reason not to; it needs the least configuration.

The exporter reads the standard OTel environment variables, so the same release runs against local, staging, and cloud backends. But read the Erlang deviations below before copying env-var snippets from another language’s guide.

OpenObserve Cloud:

export OTEL_EXPORTER_OTLP_ENDPOINT="https://cloud.openobserve.ai/api/<your-org-name>"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <your-base64-token>"
export OTEL_SERVICE_NAME="checkout-service"
export OTEL_RESOURCE_ATTRIBUTES="service.version=1.4.2,deployment.environment=production"

_build/prod/rel/checkout/bin/checkout foreground

Self-hosted OpenObserve:

export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:5080/api/default"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic cm9vdEBleGFtcGxlLmNvbTpDb21wbGV4cGFzcyMxMjM="
export OTEL_SERVICE_NAME="checkout-service"
export OTEL_RESOURCE_ATTRIBUTES="service.version=1.4.2,deployment.environment=dev"

rebar3 shell

Rules and Erlang-specific deviations that prevent most setup failures:

1. No trailing slash on the endpoint, and the organization is part of the path. The exporter appends v1/traces to the path of OTEL_EXPORTER_OTLP_ENDPOINT itself, so http://localhost:5080/api/default becomes http://localhost:5080/api/default/v1/traces. Replace default with your org name in OpenObserve Cloud (the Data Sources page shows the exact value).
2. Use a literal space in the headers value — not %20. This is the big deviation from other SDKs. The OTel spec (and guides for Go, Python, etc.) percent-encode the space as Authorization=Basic%20<token>, but the Erlang exporter parses OTEL_EXPORTER_OTLP_HEADERS by splitting on , and = without percent-decoding. Write Authorization=Basic <token> (quoted for your shell). If you send Basic%20..., OpenObserve receives the %20 literally and returns 401.
3. Signal-specific endpoint variables take the path as-is. If you set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT instead of the general variable, no suffix is appended — you must give the full .../api/default/v1/traces URL yourself.

The sys.config equivalent (shown in the setup section above) uses otlp_endpoint and otlp_headers under the opentelemetry_exporter app, with headers as {Name, Value} tuples containing a normal space — no encoding games there either. OS environment variables override the application environment, which is exactly what you want: bake sane defaults into sys.config.src and let operations override per environment.

For gRPC instead of HTTP:

{opentelemetry_exporter,
 [{otlp_protocol, grpc},
  {otlp_endpoint, "http://localhost:5081"},
  {otlp_headers, [{"authorization", "Basic cm9vdEBleGFtcGxlLmNvbTpDb21wbGV4cGFzcyMxMjM="},
                  {"organization", "default"}]}]}

For a broader tour of exporter configuration, see A Beginner’s Guide to OTLP Exporters.

Verifying data arrival in OpenObserve

Start the node, exercise some instrumented code (call checkout:process_order(<<"ord-1">>) from the shell a few times), then open the OpenObserve UI (http://localhost:5080 self-hosted, or your cloud URL):

1. Streams — a default traces stream should exist within seconds of the first batch export (the batch processor flushes every 5 s by default). If the stream exists, ingestion works end to end.
2. Traces — pick checkout-service on the Traces page and open a trace. You should see process-order with calculate-total and charge-payment nested beneath it, plus your order.id and cart.items attributes and the payment.captured event.
3. Logs — if you ship JSON logs via a collector as described above, query the log stream and filter by an otel_trace_id copied from a trace to confirm correlation works.

If nothing appears within ~30 seconds, force a flush from the shell with otel_tracer_provider:force_flush(). and watch the node’s log output — the exporter logs export failures (including HTTP status codes) through OTP logger, and that message almost always names the problem. Then work through Troubleshooting below.

Production tips

• Keep the batch processor and tune it via app env, not code. Defaults are 5 s scheduled delay, 2048-span max queue. The knobs live under the opentelemetry app: bsp_scheduled_delay_ms, bsp_max_queue_size, bsp_exporting_timeout_ms. Raise the queue only if you see drops during traffic spikes; a bigger queue is more memory on your node.

• Sample at the head, keep parent decisions authoritative. Either via environment (OTEL_TRACES_SAMPLER=parentbased_traceidratio, OTEL_TRACES_SAMPLER_ARG=0.1) or sys.config:

  {sampler, {parent_based, #{root => {trace_id_ratio_based, 0.10}}}}

  The parent_based wrapper makes your Erlang service honor sampling decisions made upstream, so distributed traces arrive complete instead of fragmented.

• Invest in resource attributes. service.name, service.version, and deployment.environment are the dimensions you filter by in every OpenObserve query. Set defaults in the resource app env and let OTEL_SERVICE_NAME / OTEL_RESOURCE_ATTRIBUTES override per environment. If you leave service.name unset you get unknown_service, and every dashboard suffers.

• Release ordering is part of correctness. opentelemetry_exporter and opentelemetry must boot before your application, or spans created during your app’s startup vanish into the no-op API. Keep them explicit in the relx release list, marked temporary so a permanent telemetry failure can never cycle the node.

• Treat context handoff as a design rule. Every spawn, gen_server message, and job queue hop is a context boundary. Standardize on passing otel_ctx:get_current() alongside the work and attaching it in the receiver — retrofitting this later is far more painful than doing it as you go.

• Don’t span the truly hot paths. Span start/end costs microseconds, which is fine for request-scoped work but not for a function called a million times a second inside a tight loop. Instrument at the boundaries (request, job, external call), not around every function.

• Upgrade the OTel apps together. opentelemetry_api, opentelemetry, and opentelemetry_exporter are released in lockstep; the experimental packages move independently and can break between 0.x versions. Pin versions in rebar.lock review and read the changelog before bumping, especially for anything experimental.

Troubleshooting

Symptom: 401 Unauthorized on every export even though the token is correct. Fix: this is almost always the percent-encoding trap. In OTEL_EXPORTER_OTLP_HEADERS the Erlang exporter does not decode %20, so use a literal space: Authorization=Basic <token>, quoted for your shell. Also regenerate the token with echo -n (a trailing newline corrupts it), and in sys.config make sure the header is a {"authorization", "Basic <token>"} tuple, not a single concatenated string.

Symptom: 404 Not Found from the exporter and nothing in OpenObserve. Fix: URL problem. OTEL_EXPORTER_OTLP_ENDPOINT must include the organization path (/api/default) and should have no trailing slash — the exporter appends v1/traces itself. If you used OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, remember it is taken as-is: it must end in /v1/traces explicitly.

Symptom: spans never appear, no errors anywhere, and ?with_span seems to do nothing. Fix: the SDK never started, so the API is running in no-op mode. In a rebar3 shell, check application:which_applications() for opentelemetry; in a release, confirm opentelemetry_exporter and opentelemetry are in the relx app list before your app. Also check the boot log for the line OTLP exporter successfully initialized — if exporter init failed, the SDK logs it once at startup and then drops spans silently.

Symptom: switching to gRPC fails to export or connect. Fix: grpcbox comes in automatically as a transitive dependency of opentelemetry_exporter, so a missing dependency is not the cause — check the endpoint and headers instead. When using gRPC against self-hosted OpenObserve, point at port 5081 (not 5080) and include the organization header next to authorization. If you do not specifically need gRPC, stay on the default http_protobuf transport.

Symptom: traces are fragmented — every process or downstream service starts a new trace. Fix: context is not crossing the boundary. Within the node, pass otel_ctx:get_current() into spawned processes and otel_ctx:attach/1 it before creating spans. Across HTTP, inject headers with otel_propagator_text_map:inject/1 on outgoing requests and extract on incoming ones (or use opentelemetry_cowboy/opentelemetry_elli, which extract for you).

Symptom: short-lived scripts or escript-style runs lose the last few spans. Fix: the batch processor’s final buffer was never exported before the VM halted. Call otel_tracer_provider:force_flush() before terminating, or stop the node gracefully (init:stop/0) so the SDK’s supervision tree shuts down its processors and flushes.

With the API in your app, the SDK and exporter riding in the release, and OTLP pointed at OpenObserve, an Erlang node gets stable, spec-compliant distributed tracing today — with trace-correlated logs via logger metadata, and metrics ready to switch on the moment the experimental packages ship their OTLP exporters.