openqa-log-analyzer · diff
git:20260505.0c4259b to git:20260511.7f16ab3
113 added, 22 removed. Audit A to A.
---
name: openqa-log-analyzer
- description: A specialized skill for analyzing openQA logs, listing test modules, and extracting specific module logs for debugging failures.
+ description: A specialized skill for analyzing openQA logs — triage failures, measure command timing, detect infrastructure lag, extract command output, and compare runs across multiple log files.
---
- # openQA Log Analyzer Skill
+ <instructions>
+ You help an OSADO developer analyze openQA log files to debug test failures.
+ Use the Perl helpers in `scripts/` to do the work. Never duplicate their logic
+ in shell, grep, or your own reasoning.
- This skill provides specialized capabilities for analyzing openQA log files (typically `autoinst-log.txt`). It leverages the project's `DEBUG_CRASH/extract_log_section.sh` script to parse the massive log files into manageable chunks.
+ Script paths below are relative to this skill's installed directory.
- ## Capabilities
+ ## Log file types
- 1. **List Modules**: Identify which test modules ran in a specific log file.
- 2. **Extract Module Logs**: Extract the full log output for a specific test module to a separate file for focused analysis.
+ openQA jobs produce several log files. This skill operates on two:
+ | Log file | Content | Used by |
+ |----------|---------|---------|
+ | `autoinst-log.txt` | Main os-autoinst log: test module lifecycle, testapi calls, serial matching, timestamps, backtraces | All scripts except `extract_cmd_output.pl` |
+ | `serial_terminal.txt` | Raw serial console I/O: command invocations and their stdout/stderr | `extract_cmd_output.pl` only |
+
+ **Locating log files:** When the user provides a directory, most scripts
+ auto-append `autoinst-log.txt`. For `extract_cmd_output.pl`, the user must
+ point to the `serial_terminal.txt` file explicitly. Typical locations:
+ - Job result directory: `testresults/<job_id>/`
+ - Downloaded logs: any user-chosen directory
+ - The `personal/` directory in the OSADO repo contains sample logs
+
## Tools & Scripts
- The primary tool is the shell script located at:
- `scripts/extract_log_section.sh`
+ | Script | Input | Purpose |
+ |--------|-------|---------|
+ | `extract_log_section.pl` | `autoinst-log.txt` | List test modules or extract a module's log section |
+ | `analyze_log_health.pl` | `autoinst-log.txt` | Quick triage: errors, timeouts, backtraces, stress warnings |
+ | `detect_lag.pl` | `autoinst-log.txt` | Correlate backend loop counts with timeouts |
+ | `measure_cmd_time.pl` | `autoinst-log.txt` | Per-command execution timing with filtering |
+ | `extract_cmd_output.pl` | `serial_terminal.txt` | Extract command stdout/stderr by regex |
+ | `compare_modules_time.pl` | Multiple `autoinst-log.txt` | Side-by-side module wall/cmd/overhead comparison |
- ### Usage
+ All scripts accept `--help`, `--json`, `--verbose`, and `--color`.
- **1. List available modules in a log file:**
+ ## Workflow
+ When asked to analyze a failure or log file, follow this pipeline:
+
+ ### Step 1 — Locate log files
+
+ Confirm the path to the log directory or files. If the user provides a
+ directory, check for both `autoinst-log.txt` and `serial_terminal.txt`.
+
+ ### Step 2 — Quick triage
+
+ Run the health check to surface all critical issues at once:
```bash
- .gemini/skills/openqa-log-analyzer/scripts/extract_log_section.sh <path_to_log_file_or_dir>
+ perl scripts/analyze_log_health.pl --json <autoinst-log.txt>
```
- *Example:* `.gemini/skills/openqa-log-analyzer/scripts/extract_log_section.sh DEBUG_CRASH/FAIL/`
+ This answers: *"What went wrong?"* — errors, timeouts, hook failures,
+ backtraces, backend stress warnings.
- **2. Extract a specific module's log:**
+ ### Step 3 — Infrastructure check
+ If timeouts were detected, check whether they were caused by host stress:
```bash
- .gemini/skills/openqa-log-analyzer/scripts/extract_log_section.sh <path_to_log_file_or_dir> <module_name>
+ perl scripts/detect_lag.pl --json <autoinst-log.txt>
```
- *Example:* `.gemini/skills/openqa-log-analyzer/scripts/extract_log_section.sh DEBUG_CRASH/FAIL/ Crash_site_a-primary`
- *Output:* Creates a file named `<module_name>.log` (e.g., `Crash_site_a-primary.log`) in the current directory.
+ This answers: *"Was this a real failure or a flaky host?"* High loop counts
+ (>100K) indicate infrastructure stress rather than SUT failure.
- ## Workflow for Analysis
+ ### Step 4 — List modules
- When asked to analyze a failure or a log file:
+ See the sequence of test modules that ran:
+ ```bash
+ perl scripts/extract_log_section.pl --json <autoinst-log.txt>
+ ```
- 1. **Locate the log file**: Confirm the path to the `autoinst-log.txt` or similar log file.
- 2. **List Modules**: Run the script without a module name to see the sequence of tests and identify the failing module (or the one of interest).
- 3. **Extract Log**: Run the script with the specific module name to generate a focused log file.
- 4. **Analyze**: Read the content of the extracted `<module_name>.log` to find errors, failures, or timeout messages.
+ ### Step 5 — Extract module log
+
+ Focus on a specific module's log section:
+ ```bash
+ perl scripts/extract_log_section.pl <autoinst-log.txt> <module_name>
+ ```
+ This creates `<module_name>.log` in the current directory.
+
+ ### Step 6 — Command timing analysis
+
+ Identify slow commands or filter by duration/API:
+ ```bash
+ perl scripts/measure_cmd_time.pl --json <autoinst-log.txt>
+ perl scripts/measure_cmd_time.pl --json --duration ">5" <autoinst-log.txt>
+ perl scripts/measure_cmd_time.pl --json <autoinst-log.txt> zypper
+ ```
+ This answers: *"Which command was slow?"*
+
+ ### Step 7 — Command output extraction (if needed)
+
+ When you need to see what a specific command actually printed:
+ ```bash
+ perl scripts/extract_cmd_output.pl --json <serial_terminal.txt> "az vm create"
+ ```
+ This answers: *"What did the command output?"*
+
+ ### Step 8 — Compare runs (if needed)
+
+ When comparing a failing run against a passing one:
+ ```bash
+ perl scripts/compare_modules_time.pl --json <pass_log.txt> <fail_log.txt>
+ ```
+ This answers: *"Which module regressed?"* or *"Where is the overhead?"*
+
+ ## Common flags
+
+ | Flag | Behavior |
+ |------|----------|
+ | `--json` | Structured JSON output (for programmatic analysis) |
+ | `--color` | ANSI-colored terminal output (for human reading) |
+ | `--verbose` | Extra progress/debug messages on stderr |
+ | `--help` | Show full usage with examples |
+ | `--duration EXPR` | Duration filter (`measure_cmd_time.pl` only). Supports `>`, `>=`, `<`, `<=`, `==`, `!=` with `&&` and `||` connectives. Example: `">1 && <=30"` |
+ | `--loop-threshold N` | Loop count threshold (`analyze_log_health.pl` only, default: 100000) |
+
+ `--json` and `--color` are mutually exclusive. Default is plain text (no color).
+
+ ## Rules
+
+ * Do NOT re-implement the helpers' logic in shell, grep, awk, or file
+ exploration tools. Always call the Perl scripts.
+ * Always prefer `--json` when processing results programmatically. Use the
+ JSON output to extract specific fields rather than parsing tabular text.
+ * Do NOT modify any log files. This skill only reads and reports.
+ * When `analyze_log_health.pl` already identified the failing module, pass
+ that module name directly to `extract_log_section.pl` rather than listing
+ all modules first.
+ * When analyzing a timeout, always run `detect_lag.pl` to rule out
+ infrastructure issues before investigating the test logic.
+ </instructions>