git:20251215.f8da749 to git:20251220.6c081e5

18 added, 276 removed. Audit A to A.

---

name: data-processing

description: "Process JSON with jq and YAML/TOML with yq. Filter, transform, query structured data efficiently. Triggers on: parse JSON, extract from YAML, query config, Docker Compose, K8s manifests, GitHub Actions workflows, package.json, filter data."

+ compatibility: "Requires jq and yq CLI tools. Install: brew install jq yq (macOS)."

+ allowed-tools: "Bash Read"

---

# Data Processing

- ## Purpose

Query, filter, and transform structured data (JSON, YAML, TOML) efficiently from the command line.

## Tools

| Tool | Command | Use For |

|------|---------|---------|

| jq | `jq '.key' file.json` | JSON processing |

| yq | `yq '.key' file.yaml` | YAML/TOML processing |

- ## jq Basics

- ### Selection and Navigation

+ ## jq Essentials

```bash

# Extract single field

jq '.name' package.json

# Extract nested field

jq '.scripts.build' package.json

# Extract from array

jq '.dependencies[0]' package.json

# Extract multiple fields

jq '{name, version}' package.json

# Navigate deeply nested

jq '.data.users[0].profile.email' response.json

- ```

- ### Array Operations

- ```bash

- # Get all array elements

- jq '.users[]' data.json

- # Get specific index

- jq '.users[0]' data.json

- # Slice array

- jq '.users[0:3]' data.json # First 3 elements

- jq '.users[-2:]' data.json # Last 2 elements

- # Array length

- jq '.users | length' data.json

- # Get array of specific field

- jq '.users[].name' data.json

- # Wrap results in array

- jq '[.users[].name]' data.json

- ```

- ### Filtering with select

- ```bash

# Filter by condition

jq '.users[] | select(.active == true)' data.json

- # Multiple conditions

- jq '.users[] | select(.age > 21 and .status == "active")' data.json

- # String contains

- jq '.users[] | select(.email | contains("@gmail"))' data.json

- # Regex match

- jq '.users[] | select(.email | test("@(gmail|yahoo)"))' data.json

- # Not null check

- jq '.users[] | select(.profile != null)' data.json

- ```

- ### Transformation with map

- ```bash

# Transform each element

jq '.users | map({id, name})' data.json

- # Add computed field

- jq '.users | map(. + {full_name: (.first + " " + .last)})' data.json

- # Filter and transform

- jq '.users | map(select(.active)) | map(.email)' data.json

- # map_values for objects

- jq '.config | map_values(. * 2)' data.json

- ```

- ### Object Manipulation

- ```bash

- # Add/update field

- jq '.version = "2.0.0"' package.json

- # Delete field

- jq 'del(.devDependencies)' package.json

- # Rename key

- jq '.dependencies | to_entries | map(.key |= gsub("@"; ""))' package.json

- # Merge objects

- jq '. + {newField: "value"}' data.json

- # Update nested field

- jq '.scripts.test = "jest --coverage"' package.json

- # Conditional update

- jq 'if .version == "1.0.0" then .version = "1.0.1" else . end' package.json

- ```

- ### Aggregation

- ```bash

- # Count

+ # Count elements

jq '.users | length' data.json

- # Sum

- jq '[.items[].price] | add' data.json

- # Min/Max

- jq '[.scores[]] | min' data.json

- jq '[.scores[]] | max' data.json

- # Average

- jq '[.scores[]] | add / length' data.json

- # Group by

- jq 'group_by(.category) | map({category: .[0].category, count: length})' data.json

- # Unique values

- jq '[.users[].role] | unique' data.json

- # Sort

- jq '.users | sort_by(.created_at)' data.json

- jq '.users | sort_by(.name) | reverse' data.json

- ```

- ### Output Formatting

- ```bash

- # Pretty print

- jq '.' response.json

- # Compact output (single line)

- jq -c '.results[]' data.json

- # Raw strings (no quotes)

+ # Raw string output

jq -r '.name' package.json

- # Tab-separated output

- jq -r '.users[] | [.id, .name, .email] | @tsv' data.json

- # CSV output

- jq -r '.users[] | [.id, .name, .email] | @csv' data.json

- # URI encoding

- jq -r '.query | @uri' data.json

```

- ## yq for YAML/TOML

- ### Basic YAML Operations

+ ## yq Essentials

```bash

# Extract field

yq '.name' config.yaml

# Extract nested

yq '.services.web.image' docker-compose.yml

# List all keys

yq 'keys' config.yaml

- # Get array element

- yq '.volumes[0]' docker-compose.yml

- ```

- ### Docker Compose Queries

- ```bash

- # List all service names

+ # List all service names (Docker Compose)

yq '.services | keys' docker-compose.yml

- # Get all images

- yq '.services[].image' docker-compose.yml

- # Get environment variables for a service

- yq '.services.web.environment' docker-compose.yml

- # Find services with specific image

- yq '.services | to_entries | map(select(.value.image | contains("nginx")))' docker-compose.yml

- ```

- ### Kubernetes Manifests

- ```bash

- # Get resource name

- yq '.metadata.name' deployment.yaml

- # Get container images

+ # Get container images (K8s)

yq '.spec.template.spec.containers[].image' deployment.yaml

- # Get all labels

- yq '.metadata.labels' deployment.yaml

- # Multi-document YAML (---)

- yq eval-all '.metadata.name' manifests.yaml

- ```

- ### GitHub Actions Workflows

- ```bash

- # List all jobs

- yq '.jobs | keys' .github/workflows/ci.yml

- # Get steps for a job

- yq '.jobs.build.steps[].name' .github/workflows/ci.yml

- # Find jobs using specific action

- yq '.jobs[].steps[] | select(.uses | contains("actions/checkout"))' .github/workflows/ci.yml

- # Get all environment variables

- yq '.env' .github/workflows/ci.yml

- ```

- ### TOML Processing

- ```bash

- # Read TOML file

- yq -p toml '.dependencies' Cargo.toml

- # Convert TOML to JSON

- yq -p toml -o json '.' config.toml

- # Extract pyproject.toml dependencies

- yq -p toml '.project.dependencies[]' pyproject.toml

- ```

- ### YAML Modification

- ```bash

# Update value (in-place)

yq -i '.version = "2.0.0"' config.yaml

- # Add new field

- yq -i '.new_field = "value"' config.yaml

- # Delete field

- yq -i 'del(.old_field)' config.yaml

- # Add to array

- yq -i '.tags += ["new-tag"]' config.yaml

- # Merge YAML files

- yq eval-all 'select(fileIndex == 0) * select(fileIndex == 1)' base.yaml override.yaml

- ```

- ## Common Config Files

- ### package.json

- ```bash

- # List all dependencies

- jq '.dependencies | keys' package.json

- # Get all scripts

- jq '.scripts' package.json

- # Find outdated patterns

- jq '.dependencies | to_entries | map(select(.value | startswith("^")))' package.json

- # Extract dev dependencies

- jq '.devDependencies | keys | .[]' package.json

- ```

- ### tsconfig.json

- ```bash

- # Get compiler options

- jq '.compilerOptions' tsconfig.json

- # Check strict mode

- jq '.compilerOptions.strict' tsconfig.json

- # List paths aliases

- jq '.compilerOptions.paths' tsconfig.json

- ```

- ### ESLint/Prettier

- ```bash

- # Get enabled rules

- jq '.rules | to_entries | map(select(.value != "off"))' .eslintrc.json

- # Check prettier options

- jq '.' .prettierrc.json

- ```

- ## Advanced Patterns

- ### Combining jq with Shell

- ```bash

- # Process multiple files

- for f in *.json; do jq '.name' "$f"; done

- # Pipeline with other tools

- curl -s https://api.github.com/users/octocat | jq '.login'

- # Assign to variable

- VERSION=$(jq -r '.version' package.json)

- # Conditional logic

- jq -e '.errors | length == 0' response.json && echo "Success"

- ```

- ### Complex Transformations

- ```bash

- # Flatten nested structure

- jq '[.categories[].items[]] | flatten' data.json

- # Reshape data

- jq '.users | map({(.id | tostring): .name}) | add' data.json

- # Pivot data

- jq 'group_by(.date) | map({date: .[0].date, values: map(.value)})' data.json

- # Join arrays

- jq -s '.[0] + .[1]' file1.json file2.json

+ # TOML to JSON

+ yq -p toml -o json '.' config.toml

```

## Quick Reference

| Task | jq | yq |

|------|----|----|

| Get field | `jq '.key'` | `yq '.key'` |

| Array element | `jq '.[0]'` | `yq '.[0]'` |

| Transform | `jq 'map(.x)'` | `yq 'map(.x)'` |

| Count | `jq 'length'` | `yq 'length'` |

| Keys | `jq 'keys'` | `yq 'keys'` |

| Pretty print | `jq '.'` | `yq '.'` |

| Compact | `jq -c` | `yq -o json -I0` |

| Raw output | `jq -r` | `yq -r` |

| In-place edit | - | `yq -i` |

## When to Use

- Reading package.json dependencies

- Parsing Docker Compose configurations

- Analyzing Kubernetes manifests

- Processing GitHub Actions workflows

- Extracting data from API responses

- Filtering large JSON datasets

- Config file manipulation

- Data format conversion

+ ## Additional Resources

+ For complete pattern libraries, load:

+ - `./references/jq-patterns.md` - Arrays, filtering, transformation, aggregation, output formatting

+ - `./references/yq-patterns.md` - Docker Compose, K8s, GitHub Actions, TOML, YAML modification

+ - `./references/config-files.md` - package.json, tsconfig, eslint/prettier patterns