async-profiler-advanced · git:20260906.e6ad4bd · 2026-09-06 · sha256 7973f28d98ca72e0
async-profiler-advanced git:20260906.e6ad4bdA
Immutable. This exact content never changes and is served at /api/v1/blob/7973f28d98ca72e0.
--- name: async-profiler-advanced description: > Configure and validate async-profiler when recordings are empty, idle-heavy, truncated, permission-blocked, containerized, multi-event, or version-sensitive. Choose event weights and engines, bound collection overhead, diagnose missing stacks, and verify conversions and differentials. Does not own initial profiler selection (jfr-and-async-profiler), visual interpretation (flame-graph-analysis), or JDK Flight Recorder configuration. --- # Async-Profiler Advanced ## Purpose Choose a collection mechanism whose signal corresponds to the engineering question, prove what the runtime actually enabled, and state what the recording cannot establish. An async-profiler file is a sample of selected events under a particular stack walker, filter, rate limit, and access envelope—not a complete account of elapsed time. Async-profiler evolves quickly. Pin its release, keep the matching binary and converter, run `asprof -v`, `asprof list <pid>`, and `asprof --help`, and consult that tag's documentation. Never make a runbook depend on `master`, a historical option name, or an assumed fallback. The command reference is authored against async-profiler v4.5 for supported HotSpot targets on Linux/macOS; it is not a promise of support for native Windows, other JVM implementations, or every JDK update. Inspect the actual runtime build, container image, architecture and JVM flags; build toolchains alone may describe a different JVM. Keep profiler build requirements, target-JVM support and the converter's Java runtime separate. Do not upgrade the application, profiler or dependencies merely to make an example fit. ## Ownership boundary - Use `jfr-and-async-profiler` to choose the least-privileged first instrument. - Use this skill to configure and validate async-profiler itself. - Use `flame-graph-analysis` to reason from stack aggregates and differentials. - Use `allocation-profiling`, `concurrency-diagnostics`, and `off-heap-memory` for the owning diagnosis once the event identifies the domain. - Use `ebpf-for-jvm` when the question is system-wide or cannot be answered from the JVM process alone. ## Start with a question contract | Question | Primary event | Weight means | Major blind spot | | ------------------------------------- | ------------------------------------ | -------------------------------------------- | --------------------------------------------------------- | | Where is on-CPU work? | `cpu` or supported CPU timer | samples/CPU-event weight | off-CPU delay | | Where is elapsed waiting? | `wall` with thread identity/state | sampled elapsed residency | causality and queue ownership | | Who allocates Java heap? | `alloc` | samples or estimated bytes | retention/liveness unless explicitly selected | | Where is contended waiting? | `lock` | sampled completed wait count or duration | ongoing waits, unsupported synchronizers, non-lock queues | | What native allocation remains? | `nativemem`, then converter `--leak` | tracked count or bytes without matching free | pre-window allocations, unhooked allocators, ownership | | Which selected calls exceed a bound? | `--trace METHOD[:T]` | recorded calls or duration | instrumentation of calls below the threshold also costs | | Which PMU event co-locates with code? | named perf event | sampled hardware/software events | counter multiplexing/skid/model dependence | Write the target process, load window, event, interval/threshold, stack mode, filters, output, rate/memory limit, expected sample volume, and validation metric before collection. If versions, command/logs, event weight, workload window, or session ownership are missing, request the smallest missing evidence and provide a conditional plan. Do not invent an engine, zero-loss result, safe interval, or diagnosis from an HTML filename. Separate observations from hypotheses; for each proposed adjustment name evidence that would confirm or refute it. ## Engine selection For CPU attribution on Linux, prefer the supported perf-events engine when kernel/user stack coverage and per-thread CPU accounting matter. Use `ctimer` when perf events are unavailable and current Linux/tool support is confirmed; use `itimer` where the platform lacks the better alternatives or for compatibility. These engines do not have identical selection probability, resolution, kernel visibility, resource cost, or failure behavior. Use wall-clock collection for off-CPU residency and latency investigations. Wall mode samples eligible threads regardless of whether they are running, parked, sleeping, or blocked. `--threads` changes output grouping for non-JFR output; `--filter` changes eligible thread IDs where supported. They are different controls. Do not assert that a requested event ran. Inspect start diagnostics, available `status`/`metrics`, output event types, weights, and kernel/native frame presence. Status and JFR event class alone may not identify the CPU engine. Mark unavailable loss counters or engine evidence as unknown. Current versions may choose a fallback for a generic CPU request; explicit event requests, platforms, and releases differ. Read [Sampling engines, events, and access](references/engines-and-events.md) when choosing an engine, interpreting lock/live/native events, diagnosing access or missing stacks, or checking PMU and virtual-thread coverage. ## Access model and least privilege Dynamic attach and event acquisition are independent: - attach requires PID-namespace visibility, compatible filesystem `/tmp` view, attach enabled, target-compatible credentials, and readable/loadable profiler library/output paths; - Linux perf events additionally depend on kernel policy, capabilities, seccomp/LSM, resource limits, PMU virtualization, and symbol policy; - native/kernel symbolization requires matching binaries/build IDs and permissions beyond merely opening the event. Diagnose the failing syscall/layer. `SYS_PTRACE` is not a universal fix for `perf_event_open`; `CAP_PERFMON`, `CAP_SYS_ADMIN`, `--all-user`, or an fd-transfer helper have different scope and security consequences. Avoid privileged containers when a CPU-timer or JFR recording answers the question. Record every exception and restore it after collection. ## Sampling economics Estimate event volume before profiling: ```text CPU-like samples ~= active CPU time / interval wall candidates ~= eligible threads * duration / interval allocation events ~= allocated bytes / allocation interval trace events ~= selected invocations passing threshold ``` These are planning estimates, not guaranteed counts. Wall-clock cost can grow with eligible thread count; this matters acutely with large platform-thread populations. Batching, filters, rate limits, stack depth, memory limits, and longer intervals trade fidelity for overhead and file size. Instrumentation modes (`--trace`, native-allocation interception) incur work on intercepted calls, including calls rejected by duration/output filters. Bound a trial using total invocation rate, not just emitted events; method retransformation can also perturb compilation. Calibrate overhead against the same workload using an unprofiled control and at least two collection intensities. Compare throughput, latency distribution, CPU, allocation/GC, and dropped/lost events. “Low overhead” is not an authorization to run every event in production. ## Multi-event recordings and time JFR output is appropriate when several async-profiler event classes or synchronized JDK JFR events need one timeline. Verify which combinations the pinned release supports; conflicts, rate budgets, and output semantics change. `--jfrsync` coordinates a JDK recording with the profiler, but configuration and event replacement semantics must be checked in that release. Clock domains, timestamp preservation, chunking, conversion, and rate limits determine whether cross-event temporal claims remain valid. A collapsed stack file destroys most event metadata and timestamps. Preserve the original JFR plus exact command before conversion. ## Differential evidence A differential flame graph is descriptive evidence, not a performance test. Before/after must share workload, duration or valid normalization, warm-up state, event configuration, filters, stack semantics, build symbolization, and enough samples. Prefer paired repetitions and validate the performance outcome separately. Normalization removes unequal total-sample tint; it cannot repair changed load mix or a different probability of selecting stacks. Positive/negative colors depend on converter and argument order—verify with a synthetic folded-stack pair rather than memorizing a palette. ## Failure modes and troubleshooting | Symptom | Distinguish | Action | | ---------------------------------------------------------------- | ------------------------------------------------------------- | ------------------------------------------------------------------------------------- | | Cannot attach | PID/tmp namespace, credentials, disabled attach, library path | Prove attach separately; align namespaces/UID; use startup agent if approved | | Perf event denied | seccomp/LSM, paranoid policy, capability, PMU virtualization | Inspect actual denial; use least privilege or explicit timer/JFR alternative | | Profile unexpectedly lacks kernel frames | selected engine, user-only mode, symbol restrictions | Inspect diagnostics/event type; do not infer successful perf collection from filename | | Wall profile is huge/perturbing | eligible threads × interval, batching, depth | Filter justified roles, increase interval, bound file/rate/memory, remeasure overhead | | Mostly idle frames | expected thread population versus incident cohort | Group/filter by role/state; correlate with requests and queues | | Unknown/truncated stacks | walker support, depth/memory limits, redefinition, symbols | Reproduce with a supported alternate walker; test any version change separately | | Allocation profile finds hot allocators but heap grows elsewhere | allocation versus retention question | Switch to live/heap-dump evidence; follow `allocation-profiling` | | Differential is one-sided everywhere | sample totals, argument order, workload mismatch | Test converter on synthetic input; normalize only after comparability is proven | | No virtual-thread application frames | mounted state and tool/JDK capability | Use JFR task/context evidence; avoid carrier-as-request conclusions | ## Anti-patterns **Anti-pattern: grant capabilities until it works.** It conflates attach, event access, and symbols while expanding blast radius. Identify the failed layer, choose the least-privileged event, and make elevated access temporary and auditable. **Anti-pattern: fixed 60-second recording.** Rare behavior may need longer or trigger-based capture; high-frequency instrumentation may need seconds. Choose duration from event rate, minimum useful sample count, incident window, overhead, and storage budget. **Anti-pattern: percentages without denominators.** A 0.3% frame may be a handful of samples; wall and CPU percentages answer different questions. Report event, weight, total, interval, confidence limits where appropriate, and corroborating system metric. **Anti-pattern: latest converter over old recording without provenance.** Converter semantics and event schemas evolve. Keep original file, producer version, converter version, command, and checksum; investigate differences before replacing prior output. ## Minimum result For a command request, return the bounded command, prerequisites, event/weight interpretation, ownership/stop rule, and success check. For a diagnostic review, add: - observed evidence with artifact/window/versions and missing information; - the leading hypothesis, competing explanation, and consequence; - a proposed adjustment and a check that could confirm or refute it; - what actually ran, artifact location, and remaining limits on the conclusion. A proposed experiment is not a measured result. A successful conversion is not proof that collection captured the intended population. Read [Session, output, and conversion protocol](references/output-and-conversion.md) before providing capture/lifecycle commands, multi-event settings, native/trace instrumentation, conversion, or a differential comparison. Read [Validation cases](references/validation-cases.md) when evaluating changes to this skill; these cases are not a required incident report. ## References - [async-profiler repository](https://github.com/async-profiler/async-profiler) — release, source, supported platforms, and matching documentation. - [Profiler options](https://github.com/async-profiler/async-profiler/blob/v4.5/docs/ProfilerOptions.md) — use the document from the pinned release tag. - [Troubleshooting](https://github.com/async-profiler/async-profiler/blob/v4.5/docs/Troubleshooting.md) — official failure guidance; correlate with the installed release. - [Linux perf security](https://docs.kernel.org/admin-guide/perf-security.html) — authoritative capability and `perf_event_paranoid` model.