data-processing ยท diff
git:20251127.0964987 to git:20251215.f8da749
312 added, 15 removed. Audit A to A.
---
name: data-processing
description: "Process JSON with jq and YAML/TOML with yq. Filter, transform, query structured data efficiently. Triggers on: parse JSON, extract from YAML, query config, Docker Compose, K8s manifests, GitHub Actions workflows, package.json, filter data."
---
# Data Processing
## Purpose
Query, filter, and transform structured data (JSON, YAML, TOML) efficiently from the command line.
## Tools
| Tool | Command | Use For |
|------|---------|---------|
| jq | `jq '.key' file.json` | JSON processing |
| yq | `yq '.key' file.yaml` | YAML/TOML processing |
- ## Usage Examples
+ ## jq Basics
- ### JSON with jq
+ ### Selection and Navigation
```bash
# Extract single field
jq '.name' package.json
# Extract nested field
- jq '.dependencies | keys' package.json
+ jq '.scripts.build' package.json
- # Filter array
+ # Extract from array
+ jq '.dependencies[0]' package.json
+
+ # Extract multiple fields
+ jq '{name, version}' package.json
+
+ # Navigate deeply nested
+ jq '.data.users[0].profile.email' response.json
+ ```
+
+ ### Array Operations
+
+ ```bash
+ # Get all array elements
+ jq '.users[]' data.json
+
+ # Get specific index
+ jq '.users[0]' data.json
+
+ # Slice array
+ jq '.users[0:3]' data.json # First 3 elements
+ jq '.users[-2:]' data.json # Last 2 elements
+
+ # Array length
+ jq '.users | length' data.json
+
+ # Get array of specific field
+ jq '.users[].name' data.json
+
+ # Wrap results in array
+ jq '[.users[].name]' data.json
+ ```
+
+ ### Filtering with select
+
+ ```bash
+ # Filter by condition
jq '.users[] | select(.active == true)' data.json
- # Transform structure
- jq '.items[] | {id, name}' data.json
+ # Multiple conditions
+ jq '.users[] | select(.age > 21 and .status == "active")' data.json
+ # String contains
+ jq '.users[] | select(.email | contains("@gmail"))' data.json
+
+ # Regex match
+ jq '.users[] | select(.email | test("@(gmail|yahoo)"))' data.json
+
+ # Not null check
+ jq '.users[] | select(.profile != null)' data.json
+ ```
+
+ ### Transformation with map
+
+ ```bash
+ # Transform each element
+ jq '.users | map({id, name})' data.json
+
+ # Add computed field
+ jq '.users | map(. + {full_name: (.first + " " + .last)})' data.json
+
+ # Filter and transform
+ jq '.users | map(select(.active)) | map(.email)' data.json
+
+ # map_values for objects
+ jq '.config | map_values(. * 2)' data.json
+ ```
+
+ ### Object Manipulation
+
+ ```bash
+ # Add/update field
+ jq '.version = "2.0.0"' package.json
+
+ # Delete field
+ jq 'del(.devDependencies)' package.json
+
+ # Rename key
+ jq '.dependencies | to_entries | map(.key |= gsub("@"; ""))' package.json
+
+ # Merge objects
+ jq '. + {newField: "value"}' data.json
+
+ # Update nested field
+ jq '.scripts.test = "jest --coverage"' package.json
+
+ # Conditional update
+ jq 'if .version == "1.0.0" then .version = "1.0.1" else . end' package.json
+ ```
+
+ ### Aggregation
+
+ ```bash
+ # Count
+ jq '.users | length' data.json
+
+ # Sum
+ jq '[.items[].price] | add' data.json
+
+ # Min/Max
+ jq '[.scores[]] | min' data.json
+ jq '[.scores[]] | max' data.json
+
+ # Average
+ jq '[.scores[]] | add / length' data.json
+
+ # Group by
+ jq 'group_by(.category) | map({category: .[0].category, count: length})' data.json
+
+ # Unique values
+ jq '[.users[].role] | unique' data.json
+
+ # Sort
+ jq '.users | sort_by(.created_at)' data.json
+ jq '.users | sort_by(.name) | reverse' data.json
+ ```
+
+ ### Output Formatting
+
+ ```bash
# Pretty print
jq '.' response.json
- # Compact output
+ # Compact output (single line)
jq -c '.results[]' data.json
+
+ # Raw strings (no quotes)
+ jq -r '.name' package.json
+
+ # Tab-separated output
+ jq -r '.users[] | [.id, .name, .email] | @tsv' data.json
+
+ # CSV output
+ jq -r '.users[] | [.id, .name, .email] | @csv' data.json
+
+ # URI encoding
+ jq -r '.query | @uri' data.json
```
- ### YAML with yq
+ ## yq for YAML/TOML
+ ### Basic YAML Operations
+
```bash
- # Extract field from YAML
+ # Extract field
+ yq '.name' config.yaml
+
+ # Extract nested
+ yq '.services.web.image' docker-compose.yml
+
+ # List all keys
+ yq 'keys' config.yaml
+
+ # Get array element
+ yq '.volumes[0]' docker-compose.yml
+ ```
+
+ ### Docker Compose Queries
+
+ ```bash
+ # List all service names
yq '.services | keys' docker-compose.yml
- # Get all container images
+ # Get all images
yq '.services[].image' docker-compose.yml
- # Extract GitHub Actions job names
- yq '.jobs | keys' .github/workflows/ci.yml
+ # Get environment variables for a service
+ yq '.services.web.environment' docker-compose.yml
- # Get K8s resource names
+ # Find services with specific image
+ yq '.services | to_entries | map(select(.value.image | contains("nginx")))' docker-compose.yml
+ ```
+
+ ### Kubernetes Manifests
+
+ ```bash
+ # Get resource name
yq '.metadata.name' deployment.yaml
- # Convert YAML to JSON
- yq -o json '.' config.yaml
+ # Get container images
+ yq '.spec.template.spec.containers[].image' deployment.yaml
+
+ # Get all labels
+ yq '.metadata.labels' deployment.yaml
+
+ # Multi-document YAML (---)
+ yq eval-all '.metadata.name' manifests.yaml
```
+ ### GitHub Actions Workflows
+
+ ```bash
+ # List all jobs
+ yq '.jobs | keys' .github/workflows/ci.yml
+
+ # Get steps for a job
+ yq '.jobs.build.steps[].name' .github/workflows/ci.yml
+
+ # Find jobs using specific action
+ yq '.jobs[].steps[] | select(.uses | contains("actions/checkout"))' .github/workflows/ci.yml
+
+ # Get all environment variables
+ yq '.env' .github/workflows/ci.yml
+ ```
+
+ ### TOML Processing
+
+ ```bash
+ # Read TOML file
+ yq -p toml '.dependencies' Cargo.toml
+
+ # Convert TOML to JSON
+ yq -p toml -o json '.' config.toml
+
+ # Extract pyproject.toml dependencies
+ yq -p toml '.project.dependencies[]' pyproject.toml
+ ```
+
+ ### YAML Modification
+
+ ```bash
+ # Update value (in-place)
+ yq -i '.version = "2.0.0"' config.yaml
+
+ # Add new field
+ yq -i '.new_field = "value"' config.yaml
+
+ # Delete field
+ yq -i 'del(.old_field)' config.yaml
+
+ # Add to array
+ yq -i '.tags += ["new-tag"]' config.yaml
+
+ # Merge YAML files
+ yq eval-all 'select(fileIndex == 0) * select(fileIndex == 1)' base.yaml override.yaml
+ ```
+
+ ## Common Config Files
+
+ ### package.json
+
+ ```bash
+ # List all dependencies
+ jq '.dependencies | keys' package.json
+
+ # Get all scripts
+ jq '.scripts' package.json
+
+ # Find outdated patterns
+ jq '.dependencies | to_entries | map(select(.value | startswith("^")))' package.json
+
+ # Extract dev dependencies
+ jq '.devDependencies | keys | .[]' package.json
+ ```
+
+ ### tsconfig.json
+
+ ```bash
+ # Get compiler options
+ jq '.compilerOptions' tsconfig.json
+
+ # Check strict mode
+ jq '.compilerOptions.strict' tsconfig.json
+
+ # List paths aliases
+ jq '.compilerOptions.paths' tsconfig.json
+ ```
+
+ ### ESLint/Prettier
+
+ ```bash
+ # Get enabled rules
+ jq '.rules | to_entries | map(select(.value != "off"))' .eslintrc.json
+
+ # Check prettier options
+ jq '.' .prettierrc.json
+ ```
+
+ ## Advanced Patterns
+
+ ### Combining jq with Shell
+
+ ```bash
+ # Process multiple files
+ for f in *.json; do jq '.name' "$f"; done
+
+ # Pipeline with other tools
+ curl -s https://api.github.com/users/octocat | jq '.login'
+
+ # Assign to variable
+ VERSION=$(jq -r '.version' package.json)
+
+ # Conditional logic
+ jq -e '.errors | length == 0' response.json && echo "Success"
+ ```
+
+ ### Complex Transformations
+
+ ```bash
+ # Flatten nested structure
+ jq '[.categories[].items[]] | flatten' data.json
+
+ # Reshape data
+ jq '.users | map({(.id | tostring): .name}) | add' data.json
+
+ # Pivot data
+ jq 'group_by(.date) | map({date: .[0].date, values: map(.value)})' data.json
+
+ # Join arrays
+ jq -s '.[0] + .[1]' file1.json file2.json
+ ```
+
+ ## Quick Reference
+
+ | Task | jq | yq |
+ |------|----|----|
+ | Get field | `jq '.key'` | `yq '.key'` |
+ | Array element | `jq '.[0]'` | `yq '.[0]'` |
+ | Filter array | `jq '.[] \| select(.x)'` | `yq '.[] \| select(.x)'` |
+ | Transform | `jq 'map(.x)'` | `yq 'map(.x)'` |
+ | Count | `jq 'length'` | `yq 'length'` |
+ | Keys | `jq 'keys'` | `yq 'keys'` |
+ | Pretty print | `jq '.'` | `yq '.'` |
+ | Compact | `jq -c` | `yq -o json -I0` |
+ | Raw output | `jq -r` | `yq -r` |
+ | In-place edit | - | `yq -i` |
+
## When to Use
- Reading package.json dependencies
- Parsing Docker Compose configurations
- Analyzing Kubernetes manifests
- Processing GitHub Actions workflows
- Extracting data from API responses
- Filtering large JSON datasets
+ - Config file manipulation
+ - Data format conversion