async-profiler-advanced · diff
git:20260906.e6ad4bd to git:20260911.37b054b
12 added, 4 removed. Audit A to A.
---
name: async-profiler-advanced
description: >
Configure and validate async-profiler when recordings are empty, idle-heavy,
truncated, permission-blocked, containerized, multi-event, or version-sensitive.
Choose event weights and engines, bound collection overhead, diagnose missing stacks,
and verify conversions and differentials. Does not own initial profiler selection
(jfr-and-async-profiler), visual interpretation (flame-graph-analysis), or JDK Flight
Recorder configuration.
---
# Async-Profiler Advanced
## Purpose
Choose a collection mechanism whose signal corresponds to the engineering question, prove
what the runtime actually enabled, and state what the recording cannot establish. An
async-profiler file is a sample of selected events under a particular stack walker, filter,
rate limit, and access envelope—not a complete account of elapsed time.
Async-profiler evolves quickly. Pin its release, keep the matching binary and converter, run
`asprof -v`, `asprof list <pid>`, and `asprof --help`, and consult that tag's documentation.
Never make a runbook depend on `master`, a historical option name, or an assumed fallback.
The command reference is authored against async-profiler v4.5 for supported HotSpot targets
on Linux/macOS; it is not a promise of support for native Windows, other JVM implementations,
or every JDK update. Inspect the actual runtime build, container image, architecture and JVM
flags; build toolchains alone may describe a different JVM. Keep profiler build requirements,
target-JVM support and the converter's Java runtime separate. Do not upgrade the application,
profiler or dependencies merely to make an example fit.
## Ownership boundary
- Use `jfr-and-async-profiler` to choose the least-privileged first instrument.
- Use this skill to configure and validate async-profiler itself.
- Use `flame-graph-analysis` to reason from stack aggregates and differentials.
- Use `allocation-profiling`, `concurrency-diagnostics`, and `off-heap-memory` for the owning
diagnosis once the event identifies the domain.
- Use `ebpf-for-jvm` when the question is system-wide or cannot be answered from the JVM
process alone.
## Start with a question contract
| Question | Primary event | Weight means | Major blind spot |
| ------------------------------------- | ------------------------------------ | -------------------------------------------- | --------------------------------------------------------- |
| Where is on-CPU work? | `cpu` or supported CPU timer | samples/CPU-event weight | off-CPU delay |
| Where is elapsed waiting? | `wall` with thread identity/state | sampled elapsed residency | causality and queue ownership |
| Who allocates Java heap? | `alloc` | samples or estimated bytes | retention/liveness unless explicitly selected |
| Where is contended waiting? | `lock` | sampled completed wait count or duration | ongoing waits, unsupported synchronizers, non-lock queues |
| What native allocation remains? | `nativemem`, then converter `--leak` | tracked count or bytes without matching free | pre-window allocations, unhooked allocators, ownership |
| Which selected calls exceed a bound? | `--trace METHOD[:T]` | recorded calls or duration | instrumentation of calls below the threshold also costs |
| Which PMU event co-locates with code? | named perf event | sampled hardware/software events | counter multiplexing/skid/model dependence |
Write the target process, load window, event, interval/threshold, stack mode, filters, output,
rate/memory limit, expected sample volume, and validation metric before collection.
If versions, command/logs, event weight, workload window, or session ownership are missing,
- request the smallest missing evidence and provide a conditional plan. Do not invent an engine,
+ reuse suitable supplied artifacts and project evidence first, then request only missing facts
+ that could change the command or conclusion. Continue checks those unknowns do not block.
+ Provide a conditional plan where needed. Do not invent an engine,
zero-loss result, safe interval, or diagnosis from an HTML filename. Separate observations
from hypotheses; for each proposed adjustment name evidence that would confirm or refute it.
## Engine selection
For CPU attribution on Linux, prefer the supported perf-events engine when kernel/user stack
coverage and per-thread CPU accounting matter. Use `ctimer` when perf events are unavailable
and current Linux/tool support is confirmed; use `itimer` where the platform lacks the better
alternatives or for compatibility. These engines do not have identical selection probability,
resolution, kernel visibility, resource cost, or failure behavior.
Use wall-clock collection for off-CPU residency and latency investigations. Wall mode samples
eligible threads regardless of whether they are running, parked, sleeping, or blocked.
`--threads` changes output grouping for non-JFR output; `--filter` changes eligible thread IDs
where supported. They are different controls.
Do not assert that a requested event ran. Inspect start diagnostics, available `status`/`metrics`,
output event types, weights, and kernel/native frame presence. Status and JFR event class alone
may not identify the CPU engine. Mark unavailable loss counters or engine evidence as unknown. Current
versions may choose a fallback for a generic CPU request; explicit event requests, platforms,
and releases differ.
Read [Sampling engines, events, and access](references/engines-and-events.md) when choosing
an engine, interpreting lock/live/native events, diagnosing access or missing stacks, or
checking PMU and virtual-thread coverage.
## Access model and least privilege
Dynamic attach and event acquisition are independent:
- attach requires PID-namespace visibility, compatible filesystem `/tmp` view, attach enabled,
target-compatible credentials, and readable/loadable profiler library/output paths;
- Linux perf events additionally depend on kernel policy, capabilities, seccomp/LSM, resource
limits, PMU virtualization, and symbol policy;
- native/kernel symbolization requires matching binaries/build IDs and permissions beyond
merely opening the event.
Diagnose the failing syscall/layer. `SYS_PTRACE` is not a universal fix for
`perf_event_open`; `CAP_PERFMON`, `CAP_SYS_ADMIN`, `--all-user`, or an fd-transfer helper have
different scope and security consequences. Avoid privileged containers when a CPU-timer or
JFR recording answers the question. Record every exception and restore it after collection.
## Sampling economics
Estimate event volume before profiling:
```text
CPU-like samples ~= active CPU time / interval
wall candidates ~= eligible threads * duration / interval
allocation events ~= allocated bytes / allocation interval
trace events ~= selected invocations passing threshold
```
These are planning estimates, not guaranteed counts. Wall-clock cost can grow with eligible
thread count; this matters acutely with large platform-thread populations. Batching, filters,
rate limits, stack depth, memory limits, and longer intervals trade fidelity for overhead and
file size. Instrumentation modes (`--trace`, native-allocation interception) incur work on intercepted
calls, including calls rejected by duration/output filters. Bound a trial using total invocation
rate, not just emitted events; method retransformation can also perturb compilation.
- Calibrate overhead against the same workload using an unprofiled control and at least two
- collection intensities. Compare throughput, latency distribution, CPU, allocation/GC, and
- dropped/lost events. “Low overhead” is not an authorization to run every event in production.
+ For a new collection configuration, calibrate overhead against the affected workload/lifecycle
+ using an unprofiled control and contrasting collection intensities. Reuse applicable calibration
+ evidence; a conversion-only request needs no new capture. Compare throughput, latency distribution,
+ CPU, allocation/GC, and dropped/lost events. During live degradation, use the existing capture
+ budget and stop rule; do not delay authorized mitigation for complete calibration or a recording.
+ If the budget cannot support collection, use existing evidence or a controlled reproduction.
+ “Low overhead” is not an authorization to run every event in production.
## Multi-event recordings and time
JFR output is appropriate when several async-profiler event classes or synchronized JDK JFR
events need one timeline. Verify which combinations the pinned release supports; conflicts,
rate budgets, and output semantics change. `--jfrsync` coordinates a JDK recording with the
profiler, but configuration and event replacement semantics must be checked in that release.
Clock domains, timestamp preservation, chunking, conversion, and rate limits determine
whether cross-event temporal claims remain valid. A collapsed stack file destroys most event
metadata and timestamps. Preserve the original JFR plus exact command before conversion.
## Differential evidence
A differential flame graph is descriptive evidence, not a performance test. Before/after must
share workload, duration or valid normalization, warm-up state, event configuration, filters,
stack semantics, build symbolization, and enough samples. Prefer paired repetitions and
validate the performance outcome separately.
Normalization removes unequal total-sample tint; it cannot repair changed load mix or a
different probability of selecting stacks. Positive/negative colors depend on converter and
argument order—verify with a synthetic folded-stack pair rather than memorizing a palette.
## Failure modes and troubleshooting
| Symptom | Distinguish | Action |
| ---------------------------------------------------------------- | ------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| Cannot attach | PID/tmp namespace, credentials, disabled attach, library path | Prove attach separately; align namespaces/UID; use startup agent if approved |
| Perf event denied | seccomp/LSM, paranoid policy, capability, PMU virtualization | Inspect actual denial; use least privilege or explicit timer/JFR alternative |
| Profile unexpectedly lacks kernel frames | selected engine, user-only mode, symbol restrictions | Inspect diagnostics/event type; do not infer successful perf collection from filename |
| Wall profile is huge/perturbing | eligible threads × interval, batching, depth | Filter justified roles, increase interval, bound file/rate/memory, remeasure overhead |
| Mostly idle frames | expected thread population versus incident cohort | Group/filter by role/state; correlate with requests and queues |
| Unknown/truncated stacks | walker support, depth/memory limits, redefinition, symbols | Reproduce with a supported alternate walker; test any version change separately |
| Allocation profile finds hot allocators but heap grows elsewhere | allocation versus retention question | Switch to live/heap-dump evidence; follow `allocation-profiling` |
| Differential is one-sided everywhere | sample totals, argument order, workload mismatch | Test converter on synthetic input; normalize only after comparability is proven |
| No virtual-thread application frames | mounted state and tool/JDK capability | Use JFR task/context evidence; avoid carrier-as-request conclusions |
## Anti-patterns
**Anti-pattern: grant capabilities until it works.** It conflates attach, event access, and
symbols while expanding blast radius. Identify the failed layer, choose the least-privileged
event, and make elevated access temporary and auditable.
**Anti-pattern: fixed 60-second recording.** Rare behavior may need longer or trigger-based
capture; high-frequency instrumentation may need seconds. Choose duration from event rate,
minimum useful sample count, incident window, overhead, and storage budget.
**Anti-pattern: percentages without denominators.** A 0.3% frame may be a handful of samples;
wall and CPU percentages answer different questions. Report event, weight, total, interval,
confidence limits where appropriate, and corroborating system metric.
**Anti-pattern: latest converter over old recording without provenance.** Converter semantics
and event schemas evolve. Keep original file, producer version, converter version, command,
and checksum; investigate differences before replacing prior output.
## Minimum result
For a command request, return the bounded command, prerequisites, event/weight interpretation,
ownership/stop rule, and success check. For a diagnostic review, add:
- observed evidence with artifact/window/versions and missing information;
- the leading hypothesis, competing explanation, and consequence;
- a proposed adjustment and a check that could confirm or refute it;
- what actually ran, artifact location, and remaining limits on the conclusion.
A proposed experiment is not a measured result. A successful conversion is not proof that
collection captured the intended population.
+ If a validated recording/conversion already answers the question, report the result, its limits
+ and any revisit condition; do not invent a configuration change or another capture to finish.
Read [Session, output, and conversion protocol](references/output-and-conversion.md) before
providing capture/lifecycle commands, multi-event settings, native/trace instrumentation,
conversion, or a differential comparison. Read
[Validation cases](references/validation-cases.md) when evaluating changes to this skill;
these cases are not a required incident report.
## References
- [async-profiler repository](https://github.com/async-profiler/async-profiler) — release,
source, supported platforms, and matching documentation.
- [Profiler options](https://github.com/async-profiler/async-profiler/blob/v4.5/docs/ProfilerOptions.md) — use the document from the pinned release tag.
- [Troubleshooting](https://github.com/async-profiler/async-profiler/blob/v4.5/docs/Troubleshooting.md) — official failure guidance; correlate with the installed release.
- [Linux perf security](https://docs.kernel.org/admin-guide/perf-security.html) — authoritative
capability and `perf_event_paranoid` model.