otel-profiles ยท diff
git:20261026.7dc08b2 to git:20261105.43d7f5a
5 added, 5 removed. Audit A to A.
---
name: otel-profiles
description: OpenTelemetry profiles signal and the eBPF profiler (otelcol-ebpf-profiler, `profiling` receiver). Use when setting up continuous profiling with OpenTelemetry, deploying the eBPF profiler on Linux hosts, Docker, or Kubernetes (DaemonSet, kind, k3d), building a profiles pipeline, sending OTLP profiles to Pyroscope, Grafana Cloud Profiles, or another backend, getting flame graphs from OTel, enabling off-CPU profiling, or debugging profiler errors about tracefs, PID namespaces, missing pods, or the service.profilesSupport feature gate. Not for SDK tracing or metrics setup.
---
# OpenTelemetry Profiles
The main OpenTelemetry producer is the [eBPF profiler](https://github.com/open-telemetry/opentelemetry-ebpf-profiler), shipped as the `profiling` receiver in the [`otelcol-ebpf-profiler`](https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-ebpf-profiler) Collector distribution. It profiles every process on a Linux host with no application changes.
Maturity: the signal entered [public Alpha](https://opentelemetry.io/blog/2026/profiles-alpha/) in March 2026, the OTLP profiles proto (`v1development`) is Development, the `profiling` receiver is Development, and the Collector gate `service.profilesSupport` is alpha. Expect breaking changes; pin the profiler, Collector, and backend versions together.
Collector-side component facts (OTLP receiver/exporter profile support, the fixed `/v1development/profiles` HTTP path, `k8s_attributes`, `memory_limiter`, `filter` `profile_conditions`) live in the `otel-collector` skill's component pages; OTTL `profile`/`profilesample` contexts live in `otel-ottl`.
- `otelcol-ebpf-profiler` `0.163.2` bundles only: receivers `profiling`, `nop`; processors `memory_limiter`, `k8s_attributes`, `resource_detection`, `resource`, `transform` (plus `batch`, which does not support profiles); exporters `otlp_grpc`, `otlp_http`, `debug`, `file`, `nop`; extensions `health_check`, `pprof`, `opamp`, `offcpu`. For anything else (`filter`, the `basicauth` extension, vendor exporters), relay to a regular Collector or build a distribution (`otel-collector-builder` skill).
+ `otelcol-ebpf-profiler` `0.164.0` bundles only: receivers `profiling`, `nop`; processors `memory_limiter`, `k8s_attributes`, `resource_detection`, `resource`, `transform` (plus `batch`, which does not support profiles); exporters `otlp_grpc`, `otlp_http`, `debug`, `file`, `nop`; extensions `health_check`, `pprof`, `opamp`, `offcpu`. For anything else (`filter`, the `basicauth` extension, vendor exporters), relay to a regular Collector or build a distribution (`otel-collector-builder` skill).
- ## Version matrix (Verified 2026-10-25)
+ ## Version matrix (Verified 2026-11-05)
| Artifact | Version | What it means |
|---|---|---|
- | `otel/opentelemetry-collector-ebpf-profiler` | `0.163.0`, `0.163.1`, `0.163.2` (newest release at verification, 2026-10-21) | Pins ebpf-profiler `v0.0.202640`: `pid_namespace_translation_mode` ([#1801](https://github.com/open-telemetry/opentelemetry-ebpf-profiler/pull/1801)); bundles the `offcpu` extension. Components unchanged from `0.163.0`. |
+ | `otel/opentelemetry-collector-ebpf-profiler` | `0.163.0` to `0.163.2`, and `0.164.0` (newest release at verification, 2026-10-26) | Pins ebpf-profiler `v0.0.202640`: `pid_namespace_translation_mode` ([#1801](https://github.com/open-telemetry/opentelemetry-ebpf-profiler/pull/1801)); bundles the `offcpu` extension. Components unchanged from `0.163.0`; `0.164.0` only bumps module versions. |
| Same image, `0.162.0` | last release before the change | Pins ebpf-profiler `v0.0.202636`: boolean `pid_namespace_translation`; does not bundle `offcpu`. |
- Evidence: [Docker Hub tags](https://hub.docker.com/r/otel/opentelemetry-collector-ebpf-profiler/tags), the [`v0.163.0`](https://github.com/open-telemetry/opentelemetry-collector-releases/blob/v0.163.0/distributions/otelcol-ebpf-profiler/manifest.yaml), [`v0.163.1`](https://github.com/open-telemetry/opentelemetry-collector-releases/blob/v0.163.1/distributions/otelcol-ebpf-profiler/manifest.yaml), and [`v0.163.2`](https://github.com/open-telemetry/opentelemetry-collector-releases/blob/v0.163.2/distributions/otelcol-ebpf-profiler/manifest.yaml) distribution manifests; the [`v0.163.2` changelog](https://github.com/open-telemetry/opentelemetry-collector-releases/blob/v0.163.2/CHANGELOG.md).
+ Evidence: [Docker Hub tags](https://hub.docker.com/r/otel/opentelemetry-collector-ebpf-profiler/tags), the [`v0.163.0`](https://github.com/open-telemetry/opentelemetry-collector-releases/blob/v0.163.0/distributions/otelcol-ebpf-profiler/manifest.yaml), [`v0.163.1`](https://github.com/open-telemetry/opentelemetry-collector-releases/blob/v0.163.1/distributions/otelcol-ebpf-profiler/manifest.yaml), [`v0.163.2`](https://github.com/open-telemetry/opentelemetry-collector-releases/blob/v0.163.2/distributions/otelcol-ebpf-profiler/manifest.yaml), and [`v0.164.0`](https://github.com/open-telemetry/opentelemetry-collector-releases/blob/v0.164.0/distributions/otelcol-ebpf-profiler/manifest.yaml) distribution manifests; the [`v0.164.0` changelog](https://github.com/open-telemetry/opentelemetry-collector-releases/blob/v0.164.0/CHANGELOG.md).
Read [references/receiver.md](references/receiver.md) when setting keys beyond `samples_per_second`, enabling off-CPU or OBI correlation, or pinning a release not in this table (keys follow the pinned ebpf-profiler, not the Collector version).
## Workflow
1. **Pick the image and pin.** Use the latest release unless a needed key or component is missing from it (see the matrix). Alternative: an OCB build (`otel-collector-builder` skill) based on the [distribution manifest](https://github.com/open-telemetry/opentelemetry-collector-releases/blob/main/distributions/otelcol-ebpf-profiler/manifest.yaml).
2. **Pick the environment.**
- Linux host or plain Docker: [Run on a host](#run-on-a-host) below.
- Kubernetes: [references/kubernetes.md](references/kubernetes.md) (DaemonSet, RBAC, Kubernetes metadata, memory sizing).
- Kubernetes nodes that are themselves containers (kind, k3d, Docker-in-Docker, minikube with the docker driver): also [references/nested-containers.md](references/nested-containers.md).
3. **Wire the pipeline.** `profiles` pipeline: `profiling` receiver, `memory_limiter` first, enrichment processors, then `otlp_grpc` or `otlp_http`. Start the Collector with `--feature-gates=+service.profilesSupport`. Run `validate --config=<file> --feature-gates=+service.profilesSupport` with the target image; it catches wrong keys for the pinned version.
4. **Export and verify.** Add `debug` (`verbosity: basic`) first: each `reporter_interval` it logs `"resource profiles": N, "sample records": M`; no lines or zero counts mean the receiver is not producing (see troubleshooting). `verbosity: detailed` shows resource attributes. Then add the backend: [references/backends.md](references/backends.md) for endpoints, auth, and per-backend label behavior.
5. **Troubleshoot** by exact error string: [references/troubleshooting.md](references/troubleshooting.md).
## Run on a host
The profiler needs root or `CAP_SYS_ADMIN`/`CAP_BPF`/`CAP_PERFMON`, the host PID namespace, and a mounted tracefs. A container gets a fresh `/sys` without tracefs even when privileged, so bind-mount it:
```bash
docker run -d --name otel-ebpf-profiler --privileged --pid=host \
-v /sys/kernel/tracing:/sys/kernel/tracing \
-v "$PWD/config.yaml:/etc/otelcol/config.yaml:ro" \
- otel/opentelemetry-collector-ebpf-profiler:0.163.2 \
+ otel/opentelemetry-collector-ebpf-profiler:0.164.0 \
--config=/etc/otelcol/config.yaml --feature-gates=+service.profilesSupport
```
```yaml
receivers:
profiling:
samples_per_second: 20 # default
processors:
memory_limiter:
check_interval: 1s
limit_mib: 800
spike_limit_mib: 200
exporters:
debug:
verbosity: basic
otlp_grpc:
endpoint: profiles-backend:4317 # backend-specific, see references/backends.md
tls:
insecure: true
service:
pipelines:
profiles:
receivers: [profiling]
processors: [memory_limiter]
exporters: [otlp_grpc, debug]
```
The image is built `FROM scratch`: no shell, so debug from logs, not `exec`. Expected startup lines: `eBPF tracer loaded`, `Attached sched monitor`, `Everything is ready`.
## Known limitations
- The receiver profiles every process; it has no process, container, or namespace selection, and upstream declined to add one ([#358](https://github.com/open-telemetry/opentelemetry-ebpf-profiler/issues/358)). To keep a subset, filter on enrichment attributes with `filter` `profile_conditions`; see "Keep only some workloads" in [references/kubernetes.md](references/kubernetes.md). Sampling cost on the node stays the same; only exported volume drops.
- Linux `amd64`/`arm64`, kernel 5.10+ (checked at config validation; `no_kernel_version_check` bypasses it). Backends may lag the proto; check which OTLP profiles version they accept.
- Trace-to-profile correlation is partial: OTLP profiles can link samples to `trace_id`/`span_id`, but the profiler fills them only from a context producer (OBI via `obi_process_ctx`; the [thread-context OTEP](https://github.com/open-telemetry/opentelemetry-specification/blob/v1.62.0/oteps/profiles/4947-thread-ctx.md) is accepted, but the receiver's `thread_context` support in `v0.0.202640` is a stub). Go pprof labels are read as custom labels.