llm-aiops · diff
git:20260618.652b2b6 to git:20260925.32ce6c0
92 added, 50 removed. Audit A to A.
---
name: llm-aiops
- description: Use when working on AIOps (AI for IT Operations) tasks — incident management, root cause analysis, log parsing/anomaly detection, cloud infrastructure management, or applying LLMs to IT operations. Also use when researching LLM-based solutions for incident response, fault localization, or infrastructure-as-code.
+ description: "LLM AIOps research reference: a curated map of LLM-based approaches to IT operations — incident management, root cause analysis (RCA), log parsing and anomaly detection, infrastructure-as-code and performance diagnosis, and AIOps Q&A. Use when triaging or designing LLM-driven incident response, fault localization, or log analysis, or when surveying the field's key systems, techniques, and benchmarks. Trigger phrases: AIOps, LLM for AIOps, incident management, incident diagnosis, root cause analysis, RCA, log parsing, log anomaly detection, logging statement generation, infrastructure-as-code, IaC diagnosis, AIOps benchmark, 智能运维, 根因分析, 故障定位, 日志解析, 日志异常检测, 运维问答."
---
# LLM AIOps Reference
- ## Overview
- Curated reference applying Large Language Models to AIOps, based on the [awesome-LLM-AIOps](https://github.com/Jun-jie-Huang/awesome-LLM-AIOps) research taxonomy. Covers incident management, log analysis, and infrastructure management.
+ ## Purpose
+ A research reference library applying Large Language Models to AIOps, built on the
+ [awesome-LLM-AIOps](https://github.com/Jun-jie-Huang/awesome-LLM-AIOps) taxonomy. It
+ answers "which systems, techniques, and benchmarks exist for this operations problem, and
+ which should I pick" — it does not run scans, parse logs, or execute anything.
+
## When to Use
- - **Incidents**: triaging, diagnosing, mitigating cloud incidents
- - **RCA**: root cause analysis of system failures
- - **Logs**: parsing, anomaly detection, logging statement generation
- - **Infrastructure**: IaC, performance diagnosis, LLM training platforms
- - **AIOps QA**: domain-specific question answering for operations
- ## Taxonomy & Key Approaches
+ - **Incidents**: triaging, diagnosing, mitigating cloud incidents.
+ - **RCA**: root cause analysis of system failures.
+ - **Logs**: parsing, anomaly detection, logging statement generation.
+ - **Infrastructure**: IaC, performance diagnosis, LLM training platforms.
+ - **AIOps QA**: domain-specific question answering for operations.
+ - Trigger phrases: "LLM for AIOps", "AI for IT operations", "incident management",
+ "root cause analysis", "log parsing", "log anomaly detection", "AIOps benchmark",
+ 「智能运维」「根因分析」「故障定位」「日志解析」「日志异常检测」「运维问答」.
- ### 1. LLM for Incident Management
+ ## When NOT to Use
- | Area | Key Approaches | Notable Systems |
- |------|---------------|-----------------|
- | **Surveys & Benchmarks** | Evaluation frameworks for AIOps agents | AIOpsLab (MLSys 2025), ITBench (ICML 2025) |
- | **Incident Diagnosis** | LLM agents + diagnostic tools | D-Bot (VLDB 2024), NetAssistant (NSDI 2024), FLASH (Preprint 2024) |
- | **Incident Reporting** | Fine-tuning, CoT prompting for summarization | Oasis (FSE 2023), COLA (ICSE-SEIP 2024), MonitorAssistant (FSE 2024) |
- | **Root Cause Analysis** | Agents with tool augmentation, CoT, ICL, multi-agent | RCAgent (CIKM 2024), mABC (EMNLP 2024), Flow-of-Action (WWW 2025), OpenRCA (ICLR 2025), COCA (ICSE 2025), RCACopilot (EuroSys 2024) |
- | **Incident Mitigation** | Agent-based with troubleshooting guides, RAG | Nissist (ECAI 2024), LLexus (SIGOPS 2024), STRATUS (NeurIPS 2025) |
- | **Postmortem Analysis** | Fine-tuning for fault profiling | FaultProfIT (ICSE-SEIP 2024), FAIL (ASE 2024) |
- | **AIOps Q&A** | Instruction tuning, RAG, domain-specialized LLMs | OWL (ICLR 2024), iKnow (ASE 2025), OpsEval (Preprint 2023), MSQA (EMNLP 2023) |
+ - The user wants an **agent to actually perform** triage, log parsing, or diagnosis on
+ their live systems -> this library only names approaches; hand off to the relevant
+ operational tool rather than pretending to execute.
+ - The task is **generic LLM application design** (prompting, RAG, fine-tuning) with no
+ operations context -> this library adds nothing; answer directly.
+ - The user needs the **current state of a specific product or vendor** -> this is a
+ research snapshot of papers through 2025; do not present it as a live product comparison.
+ - The question is about **general software/agent engineering** -> use the repository's
+ coding skills instead.
- ### 2. LLM for Log Analysis
+ ## Workflow
- | Area | Key Approaches | Notable Systems |
- |------|---------------|-----------------|
- | **Log Parsing** | ICL prompting, adaptive caching, entropy + CoT merging | DivLog (ICSE 2024), LILAC (FSE 2024), LogBatcher (ASE 2024), LUNAR (FSE 2025), LibreLog (ICSE 2025) |
- | **Log Anomaly Detection** | Prompting, RAG, fine-tuning | LogGPT (Preprint 2023), LogPrompt (ICPC 2024), RAGLog (Preprint 2024), CodeAD (2025) |
- | **Logging Statement Generation** | ICL, fine-tuning, static context analysis | UniLog (ICSE 2024), SCLogger (FSE 2024), FastLog (ISSTA 2024) |
+ 1. **Classify** the request into one of four topics: incident management, log analysis,
+ infrastructure management, or technique/benchmark selection.
+ 2. **Load exactly one routing target** from the Decision Rules table below. Do not preload
+ the whole `references/` directory.
+ 3. **Answer from the loaded file**, naming the concrete systems and their venue labels as
+ given (e.g. `D-Bot (VLDB 2024)`). Do not paraphrase a system's method beyond what the
+ table states.
+ 4. **Check the pitfalls file** (`references/pitfalls-and-outlook.md`) before recommending a
+ design, so the recommendation does not repeat a known mistake.
+ 5. **State coverage limits** when the answer is not in the library instead of improvising a
+ system or benchmark.
- ### 3. LLM for Infrastructure Management
+ ## Decision Rules
- | Area | Key Approaches | Notable Systems |
- |------|---------------|-----------------|
- | **Infrastructure-as-Code** | Agent-based bug discovery, semantic checking | Unearthing IaC Checks (SOSP 2024) |
- | **LLM Training Platform** | Black-box performance diagnosis | LLMPrism (DSN 2025) |
- | **Benchmarks** | IaC code generation | IaC-Eval (NeurIPS D&B 2024) |
+ | If the question is about... | Load |
+ |---|---|
+ | Incident triage, diagnosis, reporting, mitigation, postmortem, AIOps Q&A | `references/incident-management.md` |
+ | Log parsing, log anomaly detection, logging statement generation | `references/log-analysis.md` |
+ | Infrastructure-as-code, training-platform diagnosis, IaC benchmarks | `references/infrastructure-management.md` |
+ | Which technique to use, or which benchmark to evaluate against | `references/techniques-and-benchmarks.md` |
+ | Whether a design is a known anti-pattern, or what is deployed in production | `references/pitfalls-and-outlook.md` |
- ## Common LLM Techniques Used
+ - If the request spans two topics -> load both files, answer per topic, do not merge the
+ taxonomies into one list.
+ - If the request names a system not in the library -> say it is outside this library's
+ coverage; do not invent a venue or year for it.
- - **Prompting**: ICL (In-Context Learning), CoT (Chain-of-Thought), ToT (Tree-of-Thought)
- - **Fine-tuning**: Instruction tuning, domain-specific adaptation (e.g., OWL for IT ops)
- - **Agent-based**: Tool-augmented LLMs, multi-agent collaboration, SOP-guided agents
- - **RAG**: Retrieval-Augmented Generation for incident resolution recommendations
- - **Code Synthesis**: LLM generates rule code for anomaly detection (CodeAD)
+ ## Constraints
- ## Key Benchmarks
+ - **Cite only what the library contains.** Never invent a system name, venue, or year. If a
+ publication venue is given as `Preprint`, keep it as `Preprint`.
+ - **Preserve venue labels and dates verbatim** — they are the credibility of the reference.
+ - Treat this as a **research snapshot** (papers through 2025), not a product guide. Flag
+ that publication recency is not implementation readiness.
+ - Do not fabricate benchmark numbers or accuracy figures; the library records no metrics.
+ - Do not claim the library's systems are production-deployed beyond the one sentence in
+ `references/pitfalls-and-outlook.md`.
- - **ITBench** (ICML 2025) — Diverse real-world IT automation tasks
- - **AIOpsLab** (MLSys 2025) — Holistic autonomous cloud agent evaluation
- - **OpsEval** — Comprehensive AIOps task benchmark
- - **OpenRCA** (ICLR 2025) — RCA benchmark with fault propagation awareness
- - **IaC-Eval** (NeurIPS 2024) — Infrastructure-as-Code generation benchmark
+ ## Stop Conditions
- ## Common Mistakes
+ - Stop once the classified topic has been answered from its one reference file.
+ - Do not browse the whole `references/` directory, and do not expand into open-ended web
+ research: this library is the scope.
+ - If two topic files disagree or leave a gap, state both and stop rather than reconciling
+ by assumption.
- - ❌ Treating log parsing as a solved problem — LLM-based parsers still struggle with unseen log formats; use adaptive caching (LILAC pattern)
- - ❌ Applying generic LLMs to ops without domain tuning — Domain-specific models (OWL) or RAG significantly outperform general-purpose prompting for AIOps Q&A
- - ❌ Single-agent RCA for complex microservice failures — Multi-agent systems (mABC, Flow-of-Action) with SOP guidance outperform single-agent approaches
- - ❌ Ignoring confidence estimation in RCA — Use PACE-LM style confidence calibration before acting on LLM-generated RCA results
+ ## Verification
- ## Real-World Impact
+ - [ ] The answer names only systems present in the library, with their venue labels intact.
+ - [ ] Exactly the topic files needed were loaded — not the whole directory.
+ - [ ] Any design recommendation was checked against the pitfalls file.
+ - [ ] Out-of-scope requests were declined explicitly rather than answered speculatively.
- The field has rapidly evolved from basic prompting (2023) to sophisticated multi-agent systems with tool augmentation (2024-2025), with production-grade systems like Nissist, STRATUS, and iKnow deployed in real cloud environments. Key trend: moving from single LLM calls to autonomous agent loops with verification.
+ ## Output
+
+ - **Outcome**: a grounded answer naming the relevant systems, techniques, and benchmarks
+ for the user's operations problem.
+ - **Done when**: the question is classified, the matching reference file(s) were consulted,
+ and every cited system traces to this library.
+ - **Evidence**: the named systems with venue labels, plus an explicit statement of any
+ coverage gap.
+
+ ## References
+
+ Load only when needed:
+
+ - `references/incident-management.md` — read when the question concerns incident triage,
+ diagnosis, reporting, mitigation, postmortem, RCA, or AIOps Q&A.
+ - `references/log-analysis.md` — read when the question concerns log parsing, log anomaly
+ detection, or logging statement generation.
+ - `references/infrastructure-management.md` — read when the question concerns
+ infrastructure-as-code, training-platform diagnosis, or IaC benchmarks.
+ - `references/techniques-and-benchmarks.md` — read when choosing a technique or deciding
+ which benchmark to evaluate against.
+ - `references/pitfalls-and-outlook.md` — read before recommending a design, or when asked
+ what is actually deployed and where the field is heading.