--- name: otel-collector-config description: >- Use when authoring, debugging, or reviewing an OpenTelemetry Collector (otelcol) configuration — defining receivers/processors/exporters/connectors, wiring service pipelines, choosing between the core and contrib distributions, or exporting to OTLP/Prometheus/Loki/Tempo/Mimir. Also use when a Collector starts but no data arrives, a component is rejected at startup, or metrics lack expected labels. Triggers on "otelcol", "otel collector config", "collector pipeline", "receivers/processors/exporters", "memory_limiter", "otlp exporter", "prometheusremotewrite", "spanmetrics", "tail_sampling". --- # OpenTelemetry Collector config — doctor Generate correct configs from a plain-English description, and diagnose broken ones. The Collector is a small set of concepts with a lot of sharp edges; this skill encodes the edges so the output validates on the first try. ## Mental model (get this right and everything follows) A Collector config has four component blocks plus a `service` block that wires them: ``` receivers: # where data comes IN (otlp, prometheus, hostmetrics, filelog, kafka…) processors: # what happens in the MIDDLE, in order (memory_limiter, batch, transform…) exporters: # where data goes OUT (otlp, otlphttp, debug, prometheus, prometheusremotewrite…) connectors: # bridge one pipeline's OUTPUT into another's INPUT (spanmetrics, routing…) extensions: # collector-level features, not in the data path (health_check, pprof, zpages…) service: extensions: [health_check] pipelines: traces: { receivers: [...], processors: [...], exporters: [...] } metrics: { receivers: [...], processors: [...], exporters: [...] } logs: { receivers: [...], processors: [...], exporters: [...] } telemetry: { ... } # the Collector's OWN logs/metrics ``` **The #1 rule everyone violates:** defining a component under `receivers:` / `processors:` / `exporters:` does **nothing** on its own. A component only runs if it is referenced inside a `service.pipelines.` list. A config can be "valid" yet do nothing because a pipeline was never wired. Always check the `service` block last, and confirm every component you defined is actually used (and every name used is actually defined — that one fails validation). **Pipelines are per-signal.** `traces`, `metrics`, `logs` are separate pipelines. A receiver/exporter must support the signal of the pipeline it sits in (e.g. the `prometheus` receiver is metrics-only; putting it in a `traces` pipeline fails). You may run multiple named pipelines of the same signal (`metrics/internal`, `metrics/app`). ## Workflow ### 1. Pin the three questions - **Signals?** traces / metrics / logs (any subset). - **Sources (receivers)?** app via OTLP? scrape Prometheus targets? host metrics? tail log files? Kafka? - **Destinations (exporters)?** an OTLP backend (Tempo/Jaeger/vendor)? Prometheus (pull) or remote-write (push)? Loki? Mimir? ### 2. Choose the distribution — core vs contrib (this trips people constantly) Many common components ship **only** in `otelcol-contrib`, not core `otelcol`: `prometheus` receiver, `hostmetrics`, `filelog`, `kafka`, `resourcedetection`, `transform`/`filter` (OTTL) processors, `tail_sampling`, the `spanmetrics` connector, `prometheusremotewrite`/`loki` exporters. If the config uses any of these, it needs the **contrib** image (`otel/opentelemetry-collector-contrib`) or a custom build via the OpenTelemetry Collector Builder (`ocb`). Calling for a contrib component on the core image is a startup crash, not a YAML error — flag it. ### 3. Order processors correctly (order is significant and load-bearing) Processors run **in the listed order**. The canonical safe order: ``` processors: [memory_limiter, , , batch] ``` - **`memory_limiter` goes FIRST.** It must see data before anything buffers it, otherwise it can't shed load. Putting `batch` before it defeats the purpose. - **`batch` goes LAST** (just before export) so it batches the final shape. - **`tail_sampling`** (traces) must come before `batch`, and needs whole traces — don't shard the same trace across replicas without a load-balancing exporter in front. ### 4. Pick exporters deliberately - `otlp` = OTLP over **gRPC** (default 4317). `otlphttp` = OTLP over **HTTP** (4318). Match the backend's port/protocol. - `debug` is the console exporter (use `verbosity: detailed` while debugging). The old `logging` exporter is **removed/deprecated → use `debug`**. - Metrics to Prometheus: **`prometheus`** = Collector exposes a `/metrics` endpoint for Prometheus to **scrape** (pull). **`prometheusremotewrite`** = Collector **pushes** to a remote-write endpoint (Mimir/Thanos/Cortex/Prometheus `--web.enable-remote-write-receiver`). Don't confuse pull vs push. - Loki: the dedicated `loki` exporter was **removed** from contrib — send OTLP to Loki's native OTLP endpoint via `otlphttp` (`endpoint: http://loki:3100/otlp`). - Tempo/Jaeger: send via `otlp`/`otlphttp` — the `jaeger` **exporter** is gone (Jaeger ingests OTLP natively). The `jaeger` **receiver** still exists (beta, in core+contrib) for *ingesting* legacy Jaeger-protocol spans during a migration; don't call it removed. ### 5. Always include the safety/ops basics - `memory_limiter` processor in every pipeline that can be flooded. - `health_check` extension (and wire it into `service.extensions`). - For secrets/endpoints use env expansion: `endpoint: ${env:OTLP_ENDPOINT}` (note the `env:` prefix — bare `${VAR}` is deprecated syntax). - Set `service.telemetry.logs.level` and, if needed, expose the Collector's own metrics. ### 6. Validate before declaring done Never hand back a config you haven't validated. The Collector has a built-in validator that loads and type-checks the full config without starting it: ``` otelcol validate --config=config.yaml # core otelcol-contrib validate --config=config.yaml # contrib ``` The bundled `scripts/validate.sh` wraps this (auto-detects the contrib binary, falls back to a Docker run if no local binary). If `otelcol` isn't available, say so explicitly rather than claiming the config is validated. ## Fixing an existing config — run this checklist When handed a broken or "it starts but nothing arrives" config, check in order: 1. **Unwired components** — every receiver/processor/exporter defined but not referenced in any `service.pipelines.*`? (silent no-op) 2. **Undefined references** — a name used in a pipeline that isn't defined above? (validation error) 3. **Signal mismatch** — a metrics-only receiver in a traces pipeline, etc. 4. **core vs contrib** — a contrib-only component on the core image? (crash on start) 5. **Processor order** — `batch` before `memory_limiter`? `memory_limiter` not first? 6. **pull vs push exporter** — `prometheus` where `prometheusremotewrite` was meant (or vice-versa)? 7. **Removed components** — `logging` exporter (→`debug`), `loki` exporter (→`otlphttp`), `jaeger` **exporter** (→`otlp`; the jaeger *receiver* is still fine), bare `${VAR}` env syntax (→`${env:VAR}`). 8. **Endpoint/protocol** — gRPC vs HTTP port mismatch (4317 vs 4318); `tls`/`insecure` set correctly for the environment. 9. **Endpoint binding** — receiver bound to `localhost` when traffic comes from other containers (needs `0.0.0.0`), or bound to `0.0.0.0` in a context where that's a security concern. State the diagnosis as "what was wrong → why it failed → the fix", then output the corrected config and validate it. ## The hard parts — get these right every time (see `references/advanced.md`) A capable model already handles the basics above. These six are the ones it gets *inconsistently* right; treat them as load-bearing and pull the exact patterns from `references/advanced.md`: 1. **OTTL statement form** (`transform`/`filter` processors). The function syntax itself — `set(attributes["x"], "y") where `, `delete_key(...)`, `replace_pattern(...)` — is usually recalled correctly. What isn't: there are **two forms with different path syntax**, and mixing them fails to parse. Flat list → prefix every path (`span.attributes[...]`); inside a `context: span` block → write paths bare (`attributes[...]`). Pick one per statement list. See `references/advanced.md` §1. Always set `error_mode`; regexes are RE2 (no lookahead). 2. **spanmetrics (and any connector) is wired in TWO pipelines** — as an *exporter* in the source pipeline and a *receiver* in the destination pipeline. Wiring it once silently produces no metrics (or a validation error). 3. **Sampling must sit *downstream* of metric generation.** If `tail_sampling` (or any sampler) runs before the `spanmetrics` connector, RED metrics are computed from only the surviving fraction — request rate is wrong by the sampling ratio and *nothing looks broken*. Fork instead: the ingest pipeline exports to `[spanmetrics, forward/sampled]`; a second traces pipeline receives `forward/sampled`, applies `tail_sampling`, and exports to the trace backend. spanmetrics must see 100% of spans. 4. **`resource_to_telemetry_conversion` is a trade-off, not a default-on.** Off (the default), resource attributes land only on the `target_info` series — so `sum by (service_name)` returns nothing and you must join (`... * on (job, instance) group_left(service_name) target_info`). On, it copies **every** resource attribute onto **every** series — including `k8s.pod.name`, `host.id`, `container.id`, multiplying series count per pod. Rule: enable it only after dropping the high-cardinality resource attributes; otherwise leave it off and keep `target_info: enabled: true` for the join. State which you chose and why. 5. **`tail_sampling` at >1 replica needs a `loadbalancing` tier** keyed by `traceID`, so all spans of a trace reach the same sampler instance. Tail-sampling directly behind a plain load balancer is broken sampling. `tail_sampling` goes before `batch`. 6. **Exporter reliability — and the queue key is per-exporter.** Production exporters need `retry_on_failure` *plus a queue*, but the key differs: `otlp`/`otlphttp` use `sending_queue` (add `storage: file_storage/...` when data loss is unacceptable), while **`prometheusremotewrite` uses `remote_write_queue`** and rejects `sending_queue` at startup. If validation rejects a queue key, look up that exporter's own key — do **not** conclude the exporter has no queue and ship it with retry only. Defaults drop data on a transient backend blip. Durability differs too: `remote_write_queue` is memory-only (it takes no `storage:`), so a restart loses the backlog — but that does **not** mean the exporter has no on-disk option. `prometheusremotewrite` has its own `wal:` (`directory`, `buffer_size`, `truncate_frequency`). For survive-a-restart, use `wal` here, or move the path to `otlphttp` + `sending_queue.storage` against an OTLP-capable backend. When the request touches any of these, encode the pattern explicitly rather than trusting recall — that's the difference this skill is for. ## References - `references/components.md` — curated catalog of the most-used receivers / processors / exporters / connectors with the gotcha for each, and the core-vs-contrib split. - `references/advanced.md` — the hard parts in depth: OTTL patterns, spanmetrics dual-wiring, `resource_to_telemetry_conversion`, tail_sampling load-balancing topology, exporter reliability. - `examples/` — a before/after pair (broken → corrected, validated) and `spanmetrics-config.yaml`, a validated advanced config (spanmetrics connector + OTTL + remote-write labels + retry/queue). ## Output discipline - Emit **valid YAML only** inside config blocks — no `...` placeholders that won't parse. - **Before emitting, cross-check every name in `service.pipelines` against the blocks above it.** Connectors are the usual miss: they appear in two pipelines, so it's easy to reference `forward/x` or `spanmetrics` without ever declaring it under `connectors:`. A config that needs a "one thing to fix before you deploy" footnote is a failed answer. - Comment the non-obvious lines (why `memory_limiter` is first, why this exporter). - Prefer the smallest config that satisfies the request; don't bolt on components the user didn't ask for. - If a component is contrib-only, say so and name the image/build needed.