OpenTelemetry Collector: Setup, Config & Best Practices

The OpenTelemetry Collector is the workhorse of most production telemetry pipelines: a standalone, vendor-neutral service that receives traces, metrics, and logs in a dozen formats, processes them (batching, filtering, enrichment, sampling), and exports them to one or more backends such as OpenObserve. This guide covers what the Collector is, when you actually need one, how to install and configure it, the deployment patterns that matter, and a complete working configuration for shipping data to OpenObserve.

If you are new to OpenTelemetry as a whole — the signals, OTLP, the SDKs — start with What is OpenTelemetry? A Complete Guide and come back here for the Collector specifics.

What the Collector Is — and When You Need One

An OpenTelemetry SDK inside your application can export OTLP directly to a backend. So why insert another process in the middle?

The Collector is a pipeline engine. Data flows in through receivers, through a chain of processors, and out through exporters. Because it runs outside your application, it can do work you do not want in your request path:

• Offload quickly. Applications hand telemetry to a nearby Collector in milliseconds and move on. Retries, backoff, TLS handshakes, and slow backends become the Collector’s problem, not your service’s.
• Centralize credentials and routing. API tokens live in one place. Switching backends — or fanning out to two during a migration — is a config change, not a redeploy of every service.
• Process before you pay. Drop health-check spans, debug logs, and unused metrics before they reach (and get billed by) your backend.
• Receive non-OTLP data. Prometheus scrape targets, log files, syslog, Kafka topics, StatsD — the Collector translates them all into one pipeline.

When you can skip it: honestly, for a single service, a local dev setup, or a small deployment sending modest volume straight to OpenObserve, direct SDK export works fine and is one less thing to operate. The OpenTelemetry project itself describes direct export as appropriate for development and small-scale environments. Adopt a Collector when you have multiple services, need to control cost or enforce sampling centrally, ingest non-OTLP sources, or want telemetry to survive backend outages via queuing and retries.

Core vs. Contrib: Choosing a Distribution

The project publishes several official distributions built from the same codebase:

Distribution	Binary	What’s in it
Core	otelcol	A solid baseline — otlp, hostmetrics, jaeger, kafka, prometheus, zipkin receivers; batch, memory_limiter, attributes, resource, span, probabilistic_sampler, filter processors; debug, otlp, otlp_http, file, kafka, prometheus, prometheusremotewrite, zipkin exporters; zpages, health_check, pprof extensions; forward connector. Small surface, conservative stability.
Contrib	otelcol-contrib	The full community catalog — hundreds of receivers, processors, exporters, and connectors (filelog, prometheus, kafka, filter, transform, tail_sampling, and many more).
K8s	otelcol-k8s	A curated set for Kubernetes use cases, sized between core and contrib.
Custom (OCB)	yours	Built with the OpenTelemetry Collector Builder (ocb): exactly the components you list, nothing else. Smallest binary and attack surface.

Practical guidance: start with contrib. Core is too minimal for most real pipelines — it lacks transform, filelog, tail_sampling, k8sattributes, and the long tail of contrib-only receivers/processors. Once your config stabilizes, consider building a custom distribution with OCB so production runs only the components you actually reference.

To check whether a component you need exists and how mature it is, search the OpenTelemetry Registry or the opentelemetry-collector-contrib repository README — each component documents its stability level (development, alpha, beta, stable) per signal. For a tour of the most useful contrib components and when to reach for them, see OpenTelemetry Collector Contrib: A Comprehensive Guide.

Installing the Collector

Docker

The fastest way to get a Collector running locally:

docker run --rm \
  -v $(pwd)/otel-collector.yaml:/etc/otelcol-contrib/config.yaml \
  -p 4317:4317 -p 4318:4318 \
  otel/opentelemetry-collector-contrib:latest

Ports 4317 (OTLP/gRPC) and 4318 (OTLP/HTTP) are the standard receiver ports every OpenTelemetry SDK defaults to. Pin a specific version tag in anything beyond a quick test — the Collector releases frequently and latest will move under you. Images are published to Docker Hub (otel/opentelemetry-collector-contrib) and GitHub Container Registry (ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib).

Binary and Linux packages

Every release on the opentelemetry-collector-releases page ships tarballs plus .deb and .rpm packages for each distribution and architecture. The packages install a systemd service that reads /etc/otelcol-contrib/config.yaml:

 # Example: contrib distribution on Debian/Ubuntu (amd64)
 # Replace <version> with the latest release tag from the releases page, e.g. 0.156.0
 # (asset filenames are versioned, so /latest/download/ returns 404)
curl -sSLO https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v<version>/otelcol-contrib_<version>_linux_amd64.deb
sudo dpkg -i otelcol-contrib_<version>_linux_amd64.deb
sudo systemctl status otelcol-contrib

Edit the config, then sudo systemctl restart otelcol-contrib.

Kubernetes with Helm

The official chart deploys the Collector as a DaemonSet (agent pattern) or Deployment (gateway pattern):

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install otel-collector open-telemetry/opentelemetry-collector \
  --set mode=daemonset \
  --set image.repository="otel/opentelemetry-collector-contrib"

Set mode=deployment for a gateway tier. Your pipeline config goes under the chart’s config: value, which is merged over sane defaults.

Kubernetes also has the OpenTelemetry Operator, which manages Collectors via an OpenTelemetryCollector custom resource and adds auto-instrumentation injection for pods. It is the right tool once you run many Collector instances with different configs; for a first deployment, the Helm chart is simpler and this guide sticks with it.

Configuration Anatomy

A Collector config has up to five top-level component sections plus a service section that wires them together. Defining a component does nothing by itself — it must also be referenced in a pipeline (or in service.extensions) to be active.

 # 1. Receivers: how data gets IN
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317   # default is localhost-only; bind wider deliberately
      http:
        endpoint: 0.0.0.0:4318

 # 2. Processors: what happens to data in flight
processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80         # soft cap: % of total available memory
    spike_limit_percentage: 25   # headroom above the soft cap
  resource:
    attributes:
      - key: deployment.environment.name
        value: production
        action: upsert
  batch:
    send_batch_size: 8192
    timeout: 5s

 # 3. Exporters: where data goes OUT
exporters:
  otlp_http:
    endpoint: https://api.openobserve.ai/api/your-org
    headers:
      Authorization: Basic <base64-of-email:password>
  debug:                         # prints telemetry to stdout; dev only
    verbosity: basic

 # 4. Extensions: capabilities outside the data path
extensions:
  health_check:
    endpoint: 0.0.0.0:13133      # GET /  -> 200 when healthy
  zpages: {}                     # live debug pages on localhost:55679

 # 5. Service: activate components and define pipelines
service:
  extensions: [health_check, zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp_http]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp_http]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp_http]

Things worth internalizing:

• Pipelines are per-signal. traces, metrics, and logs pipelines are independent; a component listed in a pipeline must support that signal.
• Processor order matters. Data flows through processors in exactly the order listed. Receivers and exporters, by contrast, fan in/out — order in their lists is irrelevant.
• Components can be instantiated multiple times with the type/name syntax, e.g. otlp_http/openobserve and otlp_http/other-backend, each with its own settings.
• Environment variable substitution uses ${env:VAR_NAME} (with defaults via ${env:VAR_NAME:-fallback}). Use it to keep credentials out of config files.
• Connectors are a fifth component type that acts as the exporter of one pipeline and the receiver of another — e.g. the spanmetrics connector turns spans into RED metrics. You will not need one on day one, but know they exist.

Validate a config without starting the service:

otelcol-contrib validate --config=otel-collector.yaml

Deployment Patterns: Agent vs. Gateway

There are two fundamental shapes, and mature setups usually combine them.

Agent

A Collector instance runs on every host — a DaemonSet in Kubernetes, a system service on VMs, or occasionally a sidecar per pod. Applications export to localhost (or the node IP), which keeps the network hop short and reliable.

The agent is the right place for anything host-local: tailing container log files with the filelog receiver, collecting host metrics, and enriching telemetry with Kubernetes metadata via the k8sattributes processor (which must run near the workload to resolve pod identity).

Gateway

One horizontally scaled pool of Collectors per cluster, datacenter, or region, behind a load balancer, exposing a single OTLP endpoint. The gateway is where centralized policy lives: credentials for the backend, tail-based sampling, uniform filtering and redaction, and rate control. Costs: one more tier to operate, and a bit of added latency.

One subtlety: because a load balancer spreads requests across gateway instances, spans of one trace can land on different Collectors. Anything that needs a complete trace (tail sampling) or all data for one service (span-to-metrics aggregation) requires a two-tier gateway where the first tier runs the load-balancing exporter routing by traceID or service to the second tier.

Combining them

The common production topology is agent → gateway → backend: node agents do local collection and metadata enrichment, then forward OTLP to the gateway, which applies sampling and cost controls and exports to OpenObserve with a single set of credentials. Small clusters can run gateway-only (apps export straight to the gateway Deployment); single-node setups can run agent-only.

Sending Data to OpenObserve

OpenObserve accepts OTLP natively, so the standard otlp_http exporter is all you need. Two rules from the OpenObserve docs that account for most integration failures:

1. The endpoint is your OpenObserve base URL plus the organization path: .../api/<organization> — and it must not end with a trailing slash. The exporter appends /v1/traces, /v1/metrics, and /v1/logs itself.
2. Authentication is HTTP Basic: base64-encode email:password (use an ingestion token as the password — generate one under Data sources in the OpenObserve UI rather than using your login password).

Generate the header value:

echo -n "you@example.com:your-token" | base64

Complete config: OpenObserve Cloud

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  batch:
    send_batch_size: 8192
    timeout: 5s

exporters:
  otlp_http/openobserve:
    endpoint: https://api.openobserve.ai/api/your-org-name
    headers:
      Authorization: "Basic ${env:OPENOBSERVE_TOKEN}"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp_http/openobserve]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp_http/openobserve]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp_http/openobserve]

Here OPENOBSERVE_TOKEN holds the base64 string from the command above. Your organization name and a ready-made, copy-pasteable version of this exporter block (with the token filled in) are shown in the OpenObserve UI under Data sources → Custom → OTEL Collector.

Self-hosted OpenObserve

Only the endpoint changes — the default organization on a fresh install is default, and OTLP/HTTP shares the main port 5080:

exporters:
  otlp_http/openobserve:
    endpoint: http://localhost:5080/api/default
    headers:
      Authorization: "Basic ${env:OPENOBSERVE_TOKEN}"

Replace localhost with your service hostname as appropriate (e.g. http://openobserve.observability.svc.cluster.local:5080/api/default in Kubernetes).

Two optional refinements:

• Stream name. Logs land in the default stream unless you add a stream-name: my-stream header to the exporter to route them elsewhere.
• gRPC instead of HTTP. OpenObserve also accepts OTLP/gRPC (port 5081 self-hosted) via the otlp exporter; it additionally requires an organization: your-org header, and for plain-HTTP local endpoints you must set tls: { insecure: true }. OTLP/HTTP is the simpler default — it traverses proxies and load balancers without ceremony.

Start the Collector, generate some traffic, and data should appear in OpenObserve under Streams within seconds.

Production Processors: Ordering, Batching, and Cost Control

memory_limiter first, batch last

The two processors every production pipeline needs, in a specific arrangement:

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  batch:
    send_batch_size: 8192
    send_batch_max_size: 16384
    timeout: 5s

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]   # this order, always
      exporters: [otlp_http/openobserve]

• memory_limiter goes first — immediately after the receivers. It periodically checks memory usage; above the soft limit it refuses new data (returning errors so receivers propagate backpressure to senders, which retry), and above the hard limit it forces garbage collection. If it sits later in the chain, upstream processors can balloon memory before it ever engages. Use limit_mib/spike_limit_mib instead of percentages if you prefer absolute values.
• batch goes last, just before the exporters, so it groups fully processed data into efficient export requests. Batching dramatically reduces request count and compresses better. The batch processor defaults are send_batch_size: 8192 and timeout: 200ms (whichever is reached first triggers a flush); the examples in this guide bump timeout to 5s as a common custom starting point that batches more aggressively while still bounding latency.

The general ordering rule for anything in between: drop early, enrich late. Sampling and filtering processors go right after memory_limiter so you stop spending CPU on data you will discard; resource/attributes/transform enrichment follows; batch closes the chain.

Filtering for cost control

The filter processor (contrib) drops whole spans, metrics, or log records matching OTTL conditions. Classic wins — dropping health-check spans and debug-level logs:

processors:
  filter/noise:
    error_mode: ignore
    traces:
      span:
        - attributes["url.path"] == "/healthz"
        - attributes["url.path"] == "/readyz"
    logs:
      log_record:
        - severity_number < SEVERITY_NUMBER_INFO

Every record dropped here is ingestion, storage, and query load you never pay for. For a deeper treatment of log filtering strategies at the Collector, see Filter Logs at the Source in the OTel Collector.

Transforming and redacting

The transform processor applies OTTL statements to mutate data in flight — trim oversized attributes, normalize names, scrub PII:

processors:
  transform/scrub:
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - replace_pattern(body, "\\b\\d{16}\\b", "****REDACTED****")
          - delete_key(attributes, "http.request.header.authorization")

Related single-purpose processors: resource (add/modify resource attributes, like tagging environment or region), attributes (same for span/log/metric attributes), and redaction (allowlist-based attribute scrubbing).

Scaling and Resilience

Sending queue and retries

Exporters like otlp_http ship with a sending queue and retry logic — enabled by default, but worth configuring deliberately:

exporters:
  otlp_http/openobserve:
    endpoint: https://api.openobserve.ai/api/your-org-name
    headers:
      Authorization: "Basic ${env:OPENOBSERVE_TOKEN}"
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 5000        # batches held in memory during backend slowness
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s  # give up after 5 minutes

The queue absorbs backend blips; retries with exponential backoff handle transient failures. When the queue fills, new data is rejected and the failure is counted — watch for it (below). For durability across Collector restarts, add the file_storage extension and point the queue at it (sending_queue.storage: file_storage); queued data then survives a crash or redeploy.

Scaling out

Collectors are mostly stateless, so the gateway tier scales horizontally behind a load balancer — for OTLP/HTTP any L7 balancer works. The exceptions are stateful components: tail sampling and span-metrics aggregation need all spans of a trace (or service) on one instance, which is what the load-balancing exporter in a first-tier Collector solves by consistently routing on traceID or service to a second tier. Agents scale by definition — one per node.

Watch each Collector’s own telemetry: it exposes internal metrics (Prometheus format on localhost:8888/metrics by default) including otelcol_receiver_accepted_*, otelcol_exporter_sent_*, otelcol_exporter_send_failed_*, and queue size/capacity — the four numbers that tell you whether the pipeline is keeping up.

Troubleshooting

No data arriving in OpenObserve. Bisect the pipe. First confirm data reaches the Collector: temporarily add the debug exporter (verbosity: detailed) alongside the real one — exporters: [otlp_http/openobserve, debug] — and watch stdout. If nothing prints, the problem is upstream (SDK endpoint, ports 4317/4318 reachable, receiver bound to 0.0.0.0 rather than localhost when apps connect over the network). If data prints but never lands, read the Collector logs for export errors and check the exporter endpoint.

401 Unauthorized / 403 from the exporter. The Authorization header is wrong. Rebuild it: echo -n "email:token" | base64 — the -n matters, a trailing newline corrupts the encoding. Confirm the header reads Basic <value> with exactly one space, and that the token belongs to a user in the organization named in the URL.

404 Not Found on export. Almost always a malformed endpoint: a trailing slash after the org (/api/default/ — remove it), a missing /api/<org> path segment, or /v1/traces appended manually (the exporter adds signal paths itself).

Collector OOM-killed or memory climbing. Verify memory_limiter exists and is the first processor in every pipeline. Check whether the sending queue is growing because the backend is slow or down (otelcol_exporter_queue_size vs ..._queue_capacity). Reduce queue_size, send_batch_size, or receiver concurrency, and set container memory limits so the Collector’s soft limit is computed against reality.

Data silently dropped. Look at internal metrics on :8888: nonzero otelcol_exporter_send_failed_spans (or _log_records/_metric_points) with exhausted retries means the backend rejected data; otelcol_processor_refused_* points at memory_limiter engaging; enqueue_failed counters mean the sending queue overflowed. Each counter names a different fix: backend health, memory headroom, or queue size.

Config loads but a component “does not exist.” You are running core but referencing a contrib-only component (e.g. filter, transform, filelog) — switch to the otelcol-contrib binary/image — or the component is defined in its section but the error is a typo in the service.pipelines reference. otelcol-contrib validate --config=... catches both before deploy. For live inspection, enable the zpages extension and open http://localhost:55679/debug/pipelinez to see exactly which components are wired into which pipelines, and /debug/tracez for recent span activity.