ast-grep · diff
v1.0.1 to v1.0.1
368 added, 245 removed. Audit A to A.
---
name: ast-grep
description: Use when doing structural code search and rewriting - ast-grep linting, refactoring, multi-language patterns
metadata:
author: mte90
version: 1.0.1
tags:
- ast-grep
- code-search
- linting
- refactoring
- cli
- ast
---
# ast-grep
Fast and user-friendly tool for large-scale code searching, linting, and rewriting using AST patterns.
## Overview
ast-grep (sg) is a CLI tool that searches code based on Abstract Syntax Tree patterns, similar to syntax-aware grep/sed. It supports multiple languages and can perform automated code refactoring.
- **Fast** - Written in Rust, processes code quickly
- **Polyglot** - Supports JavaScript, TypeScript, Python, Go, Rust, Java, C, C++, and more
- **Structural** - Matches code by AST patterns, not regex
- **Rewrite** - Automated code refactoring with metavariables
- - **Debug AST** - Visualize AST and CST structures
- - **Logical Queries** - Composite queries using `all`, `any`, and `not` operators
- - **AI-assisted** - Convert natural language to YAML rules
- - **Relational Search** - Find patterns using `inside`, `has`, `precedes`
---
## Installation
```bash
# Via cargo
cargo install ast-grep
# Via npm
npm install -g ast-grep
- # Via pip
- pip install ast-grep
-
# Download pre-built binary
curl -L https://github.com/ast-grep/ast-grep/releases/download/nightly/ast-grep-x86_64-unknown-linux-musl.tar.gz | tar xz
```
---
- ## CLI Commands
+ ## Worked Rules: Real Defect Classes
- ### Pattern Search
+ This section shows three real-world rules that caught actual defects. Each demonstrates why ast-grep succeeds where grep fails.
+ ### Security Rule: Unsafed SQL String Concatenation (Python)
+
+ Detects SQL queries built by string concatenation with user input — a SQL injection vulnerability.
+
+ ```yaml
+ # rules/sql-injection.yml
+ id: sql-injection-concat
+ message: "SQL query built with string concatenation — possible injection vulnerability"
+ severity: error
+ language: Python
+ rule:
+ all:
+ - pattern: '$CURSOR.execute($SQL, $$$)'
+ - has:
+ pattern: '$USER_INPUT + $SQL'
+ inside:
+ pattern: '$SQL'
+ constraints:
+ # $SQL must contain concatenation operator
+ $SQL:
+ kind: binary_expression
+ has:
+ pattern: '+'
+ severity: error
+ ```
+
+ **Why grep cannot express this:**
+ - grep matches text patterns like `+` but cannot verify the context is a SQL execute call
+ - grep cannot distinguish `user_input + "hello"` from `user_input + "SELECT * FROM users"`
+ - ast-grep uses `inside` to confirm the concatenation feeds into `execute()`, and `kind` to verify it's a binary expression
+
+ **Test command:**
```bash
- # Basic pattern search - find all console.log calls
- ast-grep run --pattern 'console.log($ARG)' --lang javascript src/
+ ast-grep scan --rule rules/sql-injection.yml src/
+ ```
- # Short form with language inference from file extensions
- ast-grep -p 'console.log($ARG)' src/
+ **What this catches:**
+ ```python
+ # BAD — flagged
+ user_id = request.GET['id']
+ query = "SELECT * FROM users WHERE id = " + user_id
+ cursor.execute(query)
- # Search with multiple arguments using multi-metavariable
- ast-grep -p 'console.log($$$ARGS)' src/
+ # OK — not flagged (parameterized query)
+ user_id = request.GET['id']
+ cursor.execute("SELECT * FROM users WHERE id = %s", [user_id])
+ ```
- # Search for function declarations
- ast-grep -p 'function $NAME($$$PARAMS) { $$$ }' --lang typescript src/
+ ---
- # Debug pattern parsing
- ast-grep -p 'console.log($A)' --lang javascript --debug-query=ast
+ ### Correctness Rule: Empty Exception Handler
- # Show context around matches
- ast-grep -p 'TODO' --context 3 src/
+ Detects `except Exception: pass` blocks that silently swallow errors — a common correctness bug.
+
+ ```yaml
+ # rules/empty-except.yml
+ id: empty-except-block
+ message: "Empty exception handler swallows errors silently"
+ severity: warning
+ language: Python
+ rule:
+ all:
+ - pattern: 'except $EXC: pass'
+ constraints:
+ # $EXC must be Exception or a specific exception type
+ $EXC:
+ kind: identifier
+ # Match 'Exception' or specific exception names
+ regex: '^Exception$|^[A-Z][a-zA-Z]*Error$'
```
- ### Rewrite Operations
+ **Why grep cannot express this:**
+ - grep pattern `except.*: pass` matches too broadly (comments, multi-line, different contexts)
+ - grep cannot verify the `pass` is the only statement in the block
+ - ast-grep uses `all` to ensure both the exception clause AND the pass statement exist together structurally
+ **Test command:**
```bash
- # Search and rewrite - add optional chaining
- ast-grep -p '$OBJ.val && $OBJ.val()' --rewrite '$OBJ.val?.()' src/
+ ast-grep scan --rule rules/empty-except.yml src/
+ ```
- # Interactive mode - confirm each replacement
- ast-grep -p '$PROP && $PROP()' -r '$PROP?.()' --interactive src/
+ **What this catches:**
+ ```python
+ # BAD — flagged
+ try:
+ process_data()
+ except Exception: pass # Silent failure!
- # Apply all rewrites without confirmation
- ast-grep -p 'var $X' -r 'let $X' --update-all src/
+ # BAD — flagged
+ try:
+ connect_db()
+ except ConnectionError:
+ pass # Still silent!
- # Output results as JSON for piping to other tools
- ast-grep -p 'import $$$' --json src/ | jq '.[] | .text'
+ # OK — not flagged (has logging)
+ try:
+ process_data()
+ except Exception as e:
+ logger.error(e)
```
- ### Linting with Rules
+ ---
- ```bash
- # Scan with a single rule file
- ast-grep scan --rule rules/no-console.yml src/
+ ### Convention Rule: `== None` vs `is None` (Python)
- # Scan with project configuration (sgconfig.yml)
- ast-grep scan --config sgconfig.yml src/
+ Detects Python code using `== None` instead of the idiomatic `is None`.
- # Scan with inline rule (no file needed)
- ast-grep scan --inline-rules '
- id: no-debugger
- language: JavaScript
+ ```yaml
+ # rules/none-comparison.yml
+ id: prefer-is-none
+ message: "Use 'is None' instead of '== None' for Pythonic code"
+ severity: warning
+ language: Python
rule:
- pattern: debugger
- ' src/
+ pattern: '$VALUE == None'
+ constraints:
+ # Exclude None comparisons in comments
+ $VALUE:
+ kind:
+ - identifier
+ - attribute
+ - call
+ not:
+ inside:
+ kind: comment
+ ```
- # Filter rules by ID pattern
- ast-grep scan --filter 'security-*' src/
+ **Why grep cannot express this:**
+ - grep pattern `== None` matches everywhere including comments and strings
+ - grep cannot distinguish `value == None` from `"x == None" in docstring`
+ - ast-grep uses `kind` constraints to match only actual comparison expressions
- # Output in SARIF format for CI integration
- ast-grep scan --format sarif src/
+ **Test command:**
+ ```bash
+ ast-grep scan --rule rules/none-comparison.yml src/
+ ```
- # Set rule severity via CLI
- ast-grep scan --error=no-console --warning=prefer-const src/
+ **What this catches:**
+ ```python
+ # BAD — flagged
+ if data == None:
+ data = []
- # GitHub Actions compatible output
- ast-grep scan --format github src/
+ # OK — not flagged
+ if data is None:
+ data = []
+
+ # OK — not flagged (in string, not actual comparison)
+ doc = "Check if value == None"
```
---
- ## Pattern Syntax
-
- ### Metavariables
-
- - `$VAR` - Matches single AST node
- - `$$$VARGS` - Matches zero or more AST nodes (variadic)
+ ## False Positives: Practical Discipline
- ### Basic Patterns
+ A bare pattern like `$X.foo()` matches thousands of hits across a codebase. This section explains how to narrow effectively.
- ```python
- # JavaScript/TypeScript
- 'console.log($MSG)' # Single argument
- 'console.log($$$ARGS)' # Multiple arguments
- 'function $NAME($$$PARAMS) { $$$ }' # Function declaration
- 'const $X = $Y' # Variable declaration
- '$X && $X()' # Conditional call
+ ### Why Patterns Blow Up
- # Python
- 'def $FUNC($$$ARGS): $$$' # Function definition
- 'class $NAME($$$BASE): $$$' # Class definition
- 'import $X from $Y' # Named import
- 'from $X import $$$' # From import
- 'for $X in $Y: $$$' # For loop
+ ```yaml
+ # BAD — matches everything
+ rule:
+ pattern: '$X.method()'
```
- ### Language Detection
+ This matches every method call because `$X` is unconstrained. You get noise, not signal.
- ```bash
- # Explicit language
- ast-grep -p 'pattern' --lang typescript src/
+ ### Narrowing Strategies
- # Auto-detect from file extension
- ast-grep -p 'pattern' src/
+ **1. Use `constraints` to restrict metavariables:**
+
+ ```yaml
+ rule:
+ pattern: '$OBJ.value'
+ constraints:
+ $OBJ:
+ kind: identifier # Only simple names, not properties
+ regex: '^data' # Names starting with 'data'
```
- ---
+ **2. Use `inside` to require context:**
- ## Rule Configuration
+ ```yaml
+ rule:
+ pattern: 'fetch($URL)'
+ inside:
+ pattern: 'useEffect(() => { $$$ }, $$$)' # Only inside React effects
+ ```
- ### YAML Rule File
+ **3. Use `follows` / `precedes` for ordering:**
```yaml
- # rules/no-console.yml
- id: no-console
- message: "Do not use console.log in production"
- severity: warning
- language: JavaScript
rule:
- pattern: console.log($ARG)
+ pattern: 'console.log($MSG)'
+ follows:
+ pattern: 'import $$$' # Only after imports (debug logs at top)
```
- ### sgconfig.yml
+ **4. Use `has` for internal structure:**
```yaml
- # sgconfig.yml
- rules:
- - id: no-var
- message: "Use let/const instead of var"
- severity: warning
- language: JavaScript
- rule:
- pattern: var $X
- fix:
- rewrite: let $X
-
- - id: prefer-const
- message: "Use const for variables not reassigned"
- severity: warning
- language: JavaScript
- rule:
- pattern: let $X = $Y
- from: $Y
- where:
- - $Y
- fix:
- rewrite: const $X = $Y
+ rule:
+ pattern: 'function $NAME($$$)'
+ has:
+ pattern: 'await $$$' # Async functions only
```
- ### Complete Rule Example
+ **5. Use `#` to trim leading context:**
+ When your pattern accidentally captures too much from the left, use `#` to anchor:
+
```yaml
- id: security-no-inner-html
- message: "Setting innerHTML can lead to XSS attacks"
- severity: error
- language: TypeScript
+ # Matches only the statement, not the preceding line
rule:
- pattern: |
- $EL.innerHTML = $SOURCE
- fix:
- rewrite: |
- $EL.textContent = $SOURCE
- scope: script
+ pattern: '# $X = $Y' # The # anchors to statement start
```
+ ### Filtering Results with `--json`
+
+ Pipe results into external filters for complex queries:
+
+ ```bash
+ # Find matches in specific directories only
+ ast-grep -p 'console.log($$$)' --json src/ | \
+ jq -r '.[] | select(.file_path | contains("component")) | .file_path'
+
+ # Count matches per file
+ ast-grep -p 'TODO' --json src/ | \
+ jq -r '.[].file_path' | sort | uniq -c | sort -rn
+ ```
+
+ ### The Discipline
+
+ **Write rules against a known-bad sample, not against the codebase.**
+
+ 1. Create a test file with the exact defect you want to catch
+ 2. Write the rule to match that file
+ 3. Add positive cases (should match) and negative cases (should not match)
+ 4. Only then run against the full codebase
+
+ This avoids tuning rules on noise and ensures they catch what you intend.
+
---
- ## Integration
+ ## When NOT to Use ast-grep
- ### GitHub Actions
+ Not every code search problem needs AST parsing. Use the right tool.
- ```yaml
- # .github/workflows/lint.yml
- name: ast-grep
- on: [push, pull_request]
- jobs:
- lint:
- runs-on: ubuntu-latest
- steps:
- - uses: actions/checkout@v4
- - name: Run ast-grep
- uses: ast-grep/ast-grep-action@latest
- with:
- args: scan --format github src/
+ ### Use grep/ripgrep Instead
+
+ | Scenario | Tool | Reason |
+ |----------|------|--------|
+ | Plain text search (comments, strings, literals) | `grep` / `rg` | Faster, simpler, no parsing overhead |
+ | Formatting/whitespace questions | `grep` | AST ignores whitespace |
+ | Simple literal patterns | `grep` | No AST needed for `TODO` or `FIXME` |
+ | Whole-file rewrites | `sed` / formatter | Safer, more predictable |
+ | Languages with poor parser support | `grep` | ast-grep may not parse correctly |
+
+ ### The Check
+
+ - **If the pattern needs to ignore syntax** (searching text anywhere, including comments) → use grep
+ - **If the pattern needs to respect syntax** (only match actual function calls, not strings containing function calls) → use ast-grep
+
+ ### Languages with Weaker Parsing
+
+ ast-grep parses most C-family languages well. Some languages have limitations:
+
+ | Language | Status | Notes |
+ |----------|--------|-------|
+ | JavaScript, TypeScript, Python, Go, Rust | ✅ Solid | Full AST support |
+ | Java, C, C++ | ✅ Good | Mature parsers |
+ | PHP, Ruby, Kotlin | ⚠️ Partial | Some edge cases |
+ | Scala, Haskell | ⚠️ Limited | Complex grammar challenges |
+ | Custom DSLs | ❌ Not supported | No parser available |
+
+ Check if your language parses correctly:
+
+ ```bash
+ ast-grep -p 'pattern' --lang python --debug-query=ast file.py
```
- ### pre-commit
+ If the query fails or produces unexpected AST nodes, the language support may be incomplete.
+ ---
+
+ ## CI Integration
+
+ ### Pre-commit Hook
+
+ Run rules on staged files before commits:
+
```yaml
# .pre-commit-config.yaml
repos:
- repo: https://github.com/ast-grep/ast-grep
- rev: stable
+ rev: v0.25.0
hooks:
- id: ast-grep
- args: ['--config', 'sgconfig.yml', 'scan']
+ args: [scan, --config, sgconfig.yml]
```
- ### VS Code Extension
+ Or invoke directly:
- Install from VS Code Marketplace: ast-grep
+ ```bash
+ #!/bin/bash
+ # .git/hooks/pre-commit
+ ast-grep scan --config sgconfig.yml --error-on-match $(git diff --cached --name-only)
+ ```
- Features:
- - Inline diagnostics
- - Quick fixes on hover
- - Search panel
+ ### CI Job (GitHub Actions)
- ---
+ Fail the build on rule violations:
- ## Use Cases
+ ```yaml
+ # .github/workflows/lint.yml
+ name: ast-grep lint
+ on: [push, pull_request]
+ jobs:
+ lint:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v4
+ - name: Install ast-grep
+ run: npm install -g ast-grep
+ - name: Scan with rules
+ run: |
+ ast-grep scan --config sgconfig.yml --error-on-match src/
+ ```
- ### Find All TODO Comments
+ The `--error-on-match` flag causes non-zero exit on any error-severity finding.
- ```bash
- ast-grep -p 'TODO' --lang typescript src/
- ```
+ ### Baseline Workflow for Existing Codebases
- ### Remove All console.log
+ Adding rules to a large existing codebase creates noise — hundreds of matches on day one. Use baselining:
+ **Day 1: Capture existing findings as baseline**
+
```bash
- ast-grep -p 'console.log($$$ARGS)' --rewrite '' --update-all src/
+ ast-grep scan --config sgconfig.yml --json src/ > baseline-findings.json
```
- ### Replace var with let/const
+ **CI: Only fail on NEW findings**
```bash
- ast-grep -p 'var $X' -r 'const $X' --lang javascript --update-all src/
+ ast-grep scan --config sgconfig.yml --json src/ | \
+ jq -S '. - (input | .)' baseline-findings.json > new-findings.json
+
+ if [ "$(jq length new-findings.json)" -gt 0 ]; then
+ echo "New issues detected:"
+ jq -r '.[] | "\(.file_path):\(.range.start.line) \(.message)"' new-findings.json
+ exit 1
+ fi
```
- ### Add Error Boundaries in React
+ **Gradual cleanup:** Fix findings incrementally, updating the baseline after each batch.
+ ---
+
+ ## CLI Commands Reference
+
+ ### Pattern Search
+
```bash
- ast-grep -p 'class $NAME extends React.Component { $$$ }' \
- --rewrite 'class $NAME extends React.Component {
- componentDidCatch(error, info) {
- console.error(error, info);
- }
- $$$ }' \
- --lang javascript --update-all src/
+ # Basic pattern search
+ ast-grep run --pattern 'console.log($ARG)' --lang javascript src/
+
+ # Short form
+ ast-grep -p 'console.log($$$ARGS)' src/
+
+ # Show context around matches
+ ast-grep -p 'TODO' --context 3 src/
```
- ### Find Unused Variables
+ ### Rewrite Operations
```bash
- # Define rule in YAML
- # rules/no-unused-vars.yml
- id: no-unused-vars
- language: TypeScript
- rule:
- pattern: |
- const $NAME = $VALUE
- where:
- - $VALUE
- scope: code
- ```
+ # Search and rewrite
+ ast-grep -p '$OBJ.val && $OBJ.val()' --rewrite '$OBJ.val?.()' src/
- ### Advanced: Find API Patterns in Class Methods
+ # Interactive mode
+ ast-grep -p '$PROP && $PROP()' -r '$PROP?.()' --interactive src/
- ```yaml
- # Find fetch calls inside useEffect
- id: fetch-in-useeffect
- language: TypeScript
- rule:
- all:
- - pattern: 'fetch($ARGS)'
- - inside:
- pattern: 'useEffect(() => { $$$ }, $$$)'
+ # Apply all without confirmation
+ ast-grep -p 'var $X' -r 'let $X' --update-all src/
```
- ### Advanced: Find Async Functions Without Try-Catch
+ ### Linting
- ```yaml
- # Find async functions that use await but lack try-catch
- id: async-without-try
- language: TypeScript
- rule:
- all:
- - pattern: 'async function $F($$$) { $$$ }'
- - has:
- pattern: 'await $EXPR'
- - not:
- inside:
- pattern: 'try { $$$ } catch { $$$ }'
- ```
+ ```bash
+ # Scan with rules file
+ ast-grep scan --rule rules/no-console.yml src/
- ### Advanced: Find Security Anti-Patterns
+ # Scan with config
+ ast-grep scan --config sgconfig.yml src/
- ```yaml
- # Detect eval() usage
- id: no-eval
+ # Inline rule
+ ast-grep scan --inline-rules '
+ id: no-debugger
language: JavaScript
- message: "eval() is dangerous - potential code injection"
- severity: error
rule:
- pattern: 'eval($ARG)'
+ pattern: debugger
+ ' src/
- # Detect hardcoded passwords
- id: hardcoded-secret
- language: Python
- message: "Hardcoded credentials detected"
- severity: error
- rule:
- pattern: |
- password = '$PASS'
+ # SARIF output for CI
+ ast-grep scan --format sarif src/
```
---
- ## Best Practices
+ ## Pattern Syntax Quick Reference
- ### Always Test Patterns First
+ ### Metavariables
- ```bash
- # Test pattern without rewriting
- ast-grep -p 'pattern' src/
+ | Syntax | Matches |
+ |--------|---------|
+ | `$VAR` | Single AST node |
+ | `$$$VARGS` | Zero or more nodes (variadic) |
- # Then run with --dry-run equivalent (no --update-all)
- ast-grep -p 'pattern' --rewrite 'new_pattern' --interactive src/
- ```
+ ### Relational Operators
- ### Use Language Flag
+ | Operator | Meaning |
+ |----------|---------|
+ | `inside` | Pattern must be inside another pattern |
+ | `has` | Pattern must contain a sub-pattern |
+ | `follows` | Pattern appears after another |
+ | `precedes` | Pattern appears before another |
+ | `all` | All sub-patterns must match |
+ | `any` | At least one sub-pattern must match |
+ | `not` | Negation |
- ```bash
- # Explicit language avoids ambiguity
- ast-grep -p '$X' --lang python src/
- ```
+ ---
- ### Use JSON Output for Scripts
+ ## Rule Configuration
- ```bash
- ast-grep -p 'console.log' --json src/ | jq '.[] | .file_path'
- ```
+ ### YAML Rule Structure
- ### Combine with Other Tools
+ ```yaml
+ id: unique-rule-id
+ message: "Human-readable message"
+ severity: error|warning|info
+ language: JavaScript|Python|Rust|...
+ rule:
+ pattern: 'code pattern with $METAVARIABLES'
+ # Optional relational operators
+ inside:
+ pattern: 'containing context'
+ constraints:
+ $METAVARIABLE:
+ kind: node_kind
+ regex: 'pattern'
+ fix:
+ rewrite: 'replacement pattern'
+ ```
- ```bash
- # Find files modified in git and search
- git diff --name-only HEAD~1 | xargs ast-grep -p 'pattern'
+ ### Project Configuration (sgconfig.yml)
- # Search and format
- ast-grep -p 'pattern' --json src/ | jq -r '.[].file_path' | xargs prettier --write
+ ```yaml
+ rules:
+ - id: no-console
+ message: "No console.log in production"
+ severity: warning
+ language: JavaScript
+ rule:
+ pattern: console.log($ARG)
```
---
- ## Do
+ ## Best Practices
- - Test patterns with `ast-grep run --pattern` before applying fixes
- - Use `--interactive` mode for important rewrites
- - Configure `sgconfig.yml` for project-wide rules
- - Use `--lang` flag to avoid ambiguity
+ 1. **Test patterns before rewriting** — Run without `--update-all` first
+ 2. **Use `--interactive` for important changes** — Confirm each replacement
+ 3. **Explicit language flag** — Avoid ambiguity with `--lang`
+ 4. **Commit before mass rewrites** — Easy rollback if needed
+ 5. **Write rules against known-bad samples** — Not against full codebase
- ## Don't
+ ---
- - Don't use `--update-all` without testing pattern first
- - Don't rely on regex when structural patterns are needed
- - Don't forget to commit before mass rewrites
+ ## Deep Dives
+ For advanced topics, load these reference files on demand:
+
+ - **references/rule-authoring.md** — Deep dive into rule syntax, constraints, and advanced patterns (load when writing complex rules)
+ - **references/migration-guide.md** — ast-grep version migration notes (load when upgrading)
+
---
## References
- **ast-grep Docs**: https://ast-grep.github.io/
- **ast-grep GitHub**: https://github.com/ast-grep/ast-grep
- **Rule Schema**: https://ast-grep.github.io/reference/schema.html
- - **Tutorial**: https://ast-grep.github.io/tutorial/
+