data-analysis v1.0.0A

Immutable. This exact content never changes and is served at /api/v1/blob/c14fd78949d4b1f4.

---
name: data-analysis
description: >
  Use this skill when analyzing structured datasets: load, clean, explore,
  visualize, and report on data from CSV, JSON, Excel, or SQL sources. Produces
  statistical summaries, correlation matrices, time series analysis, regression
  models, hypothesis tests, and publication-quality visualizations.
version: 1.0.0
author: Skill Foundry
platforms:
  - claude-code
  - codex
  - cursor
  - gemini-cli
  - openclaw
  - copilot
tags:
  - data-analysis
  - visualization
  - statistics
  - pandas
  - matplotlib
  - seaborn
  - plotly
  - reporting
  - csv
  - json
  - excel
  - sql
---

# Data Analysis Agent Skill

## Overview

A production-grade agent skill for end-to-end data analysis workflows. Use this
skill when you need to load datasets, inspect structure, clean messy data,
perform statistical analyses, create visualizations, or generate structured
reports from tabular data.

## When to Trigger

Activate this skill when the user asks to:

- **Analyze data**: "Analyze this CSV", "What patterns do you see in sales.csv?",
  "Explore this dataset", "Summarize the data in users.json"
- **Create charts/graphs**: "Make a bar chart of…", "Plot revenue over time",
  "Visualize the correlation matrix", "Show me a heatmap of…"
- **Find patterns**: "Find trends in this data", "Is there a correlation between
  X and Y?", "Cluster customers from this data", "Detect anomalies in…"
- **Generate reports**: "Create a report from survey_results.xlsx",
  "Summarize quarterly metrics", "Build a dashboard from sales data"
- **Clean data**: "Clean this messy dataset", "Fix missing values in…",
  "Normalize these columns", "Deduplicate this CSV"
- **Statistical testing**: "Run a t-test on group A vs B", "Check if this
  distribution is normal", "Perform regression analysis", "Calculate
  confidence intervals"

### Near-Miss Negatives — Do NOT Trigger

- Questions about **database schema design** without actual data (e.g.,
  "What columns should my users table have?")
- Questions about **spreadsheet software UI** (e.g., "How do I freeze a row in
  Google Sheets?")
- General **math / statistics theory** questions without a dataset context
  (e.g., "Explain the central limit theorem")
- Pure **SQL query writing** without a data-analysis intent (e.g., "Write a
  query to join three tables" — use a SQL skill instead)
- Questions about **ETL pipeline architecture** or data engineering (e.g.,
  "Design a data ingestion pipeline")

## Step-by-Step Workflow

Follow these phases in order. Skip phases that don't apply (e.g., if data is
already clean) but always **state that you're skipping** and why.

### Phase 1: Load

Determine the data source and load it into a DataFrame.

```
CSV    → pd.read_csv(filepath, ...)
JSON   → pd.read_json(filepath, ...)
Excel  → pd.read_excel(filepath, sheet_name=...)
SQL    → pd.read_sql_query(query, connection)
```

**Checklist:**
- [ ] Identify encoding (try UTF-8, Latin-1, detect automatically)
- [ ] For CSV: inspect delimiter (comma, tab, semicolon), quote character
- [ ] For Excel: list available sheets, load the right one
- [ ] For JSON: handle nested structures with `pd.json_normalize()` if needed
- [ ] For SQL: confirm read-only access, never run destructive queries
- [ ] Load a sample first if the dataset is large (>100k rows)

### Phase 2: Inspect

Understand what you're working with before touching anything.

```python
df.shape          # rows × columns
df.info()         # dtypes, non-null counts, memory
df.head(10)       # first rows
df.tail(5)        # last rows
df.describe()     # numeric summary stats
df.describe(include='object')  # categorical summary
df.dtypes         # column types
df.columns.tolist()  # column names
```

**Checklist:**
- [ ] Report shape: rows × columns
- [ ] List all columns with their dtypes
- [ ] Show summary statistics for numeric columns
- [ ] Show value counts for low-cardinality categorical columns
- [ ] Flag potential issues: wrong dtypes, placeholder values, suspicious zeros

### Phase 3: Clean

Address data quality issues. **Never modify the source file** without prompting
the user first. Work on a copy.

**Common operations:**

| Issue | Approach |
|-------|----------|
| Missing values | `df.isnull().sum()` → decide drop vs impute |
| Wrong dtypes | `pd.to_numeric()`, `pd.to_datetime()`, `astype()` |
| Outliers | IQR method, Z-score, domain-specific thresholds |
| Duplicates | `df.duplicated().sum()` → `df.drop_duplicates()` |
| Inconsistent strings | `.str.strip()`, `.str.lower()`, `.str.replace()` |
| Date parsing | `pd.to_datetime()` with format or infer |
| Normalization | Min-max scaling, Z-score standardization |
| Categorical encoding | One-hot, label encoding for ML prep |

**Rules:**
1. Always work on `df_clean = df.copy()`, never mutate the original in-place
   without explicit user consent.
2. Report every change: "Dropped 47 duplicate rows (2.3% of data)", "Imputed
   missing age values with median (142 cells)".
3. Flag suspicious patterns even if you don't fix them: "Column 'salary' has
   340 zero values — verify if these are legitimate."
4. If a cleaning decision is irreversible, ask first.

### Phase 4: Analyze

Apply appropriate analytical methods based on the question.

**Exploratory Data Analysis (EDA):**
- Univariate: histograms, box plots, value counts per column
- Bivariate: scatter plots, correlation coefficients, grouped means
- Multivariate: pair plots, correlation matrix heatmap, PCA

**Statistical Methods (see `references/statistical-methods.md`):**

| Goal | Method |
|------|--------|
| Compare two groups | Independent t-test, Mann-Whitney U |
| Compare 3+ groups | One-way ANOVA, Kruskal-Wallis |
| Relationship between two continuous vars | Pearson/Spearman correlation |
| Predict continuous outcome | Linear regression, polynomial regression |
| Predict categorical outcome | Logistic regression |
| Check normality | Shapiro-Wilk test, Q-Q plot |
| Detect time trends | Moving averages, decomposition, stationarity tests |
| Find clusters | K-means, hierarchical clustering, DBSCAN |
| Reduce dimensions | PCA, t-SNE (visualization only) |

**Time Series specifics:**
- Set datetime index: `df.set_index('date', inplace=True)`
- Resample: `df.resample('M').mean()`
- Rolling windows: `df['value'].rolling(7).mean()`
- Decomposition: trend, seasonal, residual

### Phase 5: Visualize

Choose the right chart for the data and question. See
`references/visualization-patterns.md` for the full guide.

**Library selection:**
- **Static, publication-quality**: `matplotlib` + `seaborn`
- **Interactive, exploratory**: `plotly`
- **Statistical plots**: `seaborn` (box, violin, pair, joint, heatmap)

**Quick reference:**

| Data Type | Question | Chart |
|-----------|----------|-------|
| Categorical × Numeric | Compare amounts | Bar chart, box plot |
| Numeric × Numeric | Relationship | Scatter plot, line chart |
| Time × Numeric | Trend over time | Line chart, area chart |
| Categorical × Categorical | Cross-tabulation | Heatmap, stacked bar |
| Distribution | Shape of data | Histogram, KDE, violin |
| Part-to-whole | Proportions | Pie chart* (≤5 categories), treemap |
| Correlation matrix | Relationships | Heatmap |
| Rankings | Order | Horizontal bar chart |

*Pie charts: use only when ≤5 categories and values sum to a meaningful whole.
Prefer bar charts otherwise.

**Best practices:**
- Always label axes and add a title
- Use accessible color palettes (avoid red-green for colorblind users)
- Sort bar charts by value unless categories have a natural order
- Add data source and date to chart footnotes
- For interactive charts, include hover tooltips

### Phase 6: Report

Synthesize findings into a structured report.

**Report structure:**
1. **Executive Summary** — 2–3 sentences with the key finding
2. **Data Overview** — source, shape, date range, columns
3. **Data Quality** — issues found, actions taken
4. **Key Findings** — bullet points with numbers, ranked by importance
5. **Visualizations** — inline charts with captions
6. **Statistical Results** — test statistics, p-values, effect sizes
7. **Limitations & Caveats** — data gaps, assumptions, edge cases
8. **Recommendations** — actionable next steps or further analysis

**Output formats:**
- **Quick answer**: plain text summary in chat with key numbers
- **Detailed report**: Markdown document with embedded charts
- **Dashboard**: interactive HTML with plotly (offer if >5 charts)
- **Export**: offer to save cleaned data and charts as files

## Tool-Aware Implementation

### Python Libraries

This skill assumes Python 3.9+ with the following libraries available.
Check availability before use; install missing packages as needed.

```python
# Core
import pandas as pd
import numpy as np

# Visualization
import matplotlib.pyplot as plt
import seaborn as sns

# Interactive
import plotly.express as px
import plotly.graph_objects as go

# Statistics
from scipy import stats
from scipy.stats import norm, ttest_ind, f_oneway, pearsonr, spearmanr

# Optional: machine learning
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
from sklearn.linear_model import LinearRegression, LogisticRegression
```

**Matplotlib setup for non-interactive environments:**
```python
import matplotlib
matplotlib.use('Agg')  # headless rendering
```

**Plotly in notebooks vs scripts:**
```python
# In Jupyter/notebook environments:
import plotly.io as pio
pio.renderers.default = 'notebook'

# For saving to files:
fig.write_html('chart.html')
fig.write_image('chart.png')
```

### Platform-Specific Notes

| Platform | Matplotlib backend | File output | Notes |
|----------|-------------------|-------------|-------|
| Claude Code | Agg | write to files | Save charts as PNG/HTML, display from disk |
| Codex | Agg | write to files | Same approach |
| Cursor | Agg or interactive | write to files | Can open HTML in preview |
| Gemini CLI | Agg | write to files | Save charts, display paths |
| OpenClaw | Agg | write to files | Use canvas for HTML output |
| Copilot | Agg | write to files | Standard file-based approach |

## Safety & Guardrails

1. **Never modify source data in-place** — always create a copy or backup before
   transformations. Offer to save cleaned data as a new file.
2. **Flag data quality issues** — don't silently fix problems. Report missing
   values, outliers, and type inconsistencies before and after cleaning.
3. **Statistical honesty** — report p-values and effect sizes, not just
   "significant" or "not significant". Don't p-hack by running multiple tests
   without correction. Mention when sample sizes are too small for reliable
   inference.
4. **Privacy awareness** — if a dataset appears to contain PII (emails, phone
   numbers, names), warn the user and suggest anonymization before analysis.
5. **Large file handling** — for files >100MB, use chunked reading
   (`chunksize` parameter) or sample before full analysis. Warn about memory
   constraints.
6. **SQL safety** — use read-only connections. Never run INSERT, UPDATE,
   DELETE, DROP, or ALTER. Use transactions or connection strings that enforce
   read-only mode.
7. **Deterministic results** — set random seeds for reproducible analysis:
   `np.random.seed(42)`.

## Scripts

### `scripts/validate_dataset.py`

PEP 723 compliant data quality validation script. Run with:

```bash
python scripts/validate_dataset.py path/to/dataset.csv
# or
python scripts/validate_dataset.py path/to/dataset.json
# or
python scripts/validate_dataset.py path/to/dataset.xlsx --sheet "Sheet1"
```

Produces a structured quality report covering missing values, outliers, type
consistency, duplicates, and basic statistics. See script docstring for details.

## References

- **[data-cleaning-guide.md](references/data-cleaning-guide.md)** — Handling
  nulls, outliers, type coercion, deduplication, and normalization patterns.
- **[visualization-patterns.md](references/visualization-patterns.md)** — Chart
  type selection guide, color best practices, accessibility considerations.
- **[statistical-methods.md](references/statistical-methods.md)** — Descriptive
  statistics, hypothesis testing, regression, correlation, confidence intervals,
  and when to use each method.