v1.0.0 to v2.0.0
90 added, 55 removed. Audit A to A.
---
name: monte-carlo-storage-cost-analysis
- description: |
- Identifies storage waste patterns (unread tables, zombies, dead-ends) and
- recommends safe cleanup actions with cost savings estimates. Uses lineage
- to verify downstream dependencies before recommending removal. Activates
- when a user asks about storage costs, unused tables, or warehouse cleanup.
- version: 1.0.0
+ description: Analyze a warehouse for stale, unused, or redundant tables via the analyze_storage_costs MCP tool. Classifies waste patterns and table categories, computes safety tiers, and handles category drill-downs and lineage follow-ups.
+ version: 2.0.0
---
# Monte Carlo Storage Cost Analysis Skill
- This skill helps identify tables that are wasting storage and recommends safe cleanup actions. It uses Monte Carlo's observability data -- lineage, query activity, monitoring status, and importance scores -- to classify waste patterns and compute safety tiers before recommending any action.
+ This skill analyzes a data warehouse for stale tables that can be removed to reduce storage costs. It delegates classification, safety scoring, and formatting to the `analyze_storage_costs` MCP tool, then presents the pre-formatted result verbatim and handles follow-up questions (category drill-downs, lineage checks).
- Reference files live next to this skill file. **Use the Read tool** (not MCP resources) to access them:
+ Reference file (use the Read tool to access it):
- - Waste pattern definitions and classification: `references/waste-patterns.md` (relative to this file)
- - Safety tier computation: `references/safety-tiers.md` (relative to this file)
+ - Output contract and category keywords: `references/output-structure.md`
## When to activate this skill
Activate when the user:
- Asks about storage costs, waste, or cleanup opportunities
- Wants to find unused, unread, or stale tables
- Asks "which tables can I drop?" or "what's costing us money?"
- Mentions storage optimization, cost reduction, or warehouse cleanup
- - Wants to identify zombie tables or dead-end pipelines
+ - Wants to identify zombie tables, dead-end pipelines, or temporary/archive tables
## When NOT to activate this skill
Do not activate when the user is:
- Just querying data or exploring table contents
- Creating or modifying monitors (use the monitoring-advisor skill)
- Investigating data quality incidents (use the prevent skill)
- - Looking at pipeline performance (use the performance-diagnosis skill)
+ - Looking at pipeline performance or query cost (use the performance-diagnosis skill)
## Prerequisites
The following MCP tools must be available (connect to Monte Carlo's MCP server):
- - `search` -- find tables by name, filter by monitoring status and importance
- - `get_table` -- get table metadata (size, type, timestamps)
- - `get_asset_lineage` -- check upstream/downstream dependencies
- - `get_queries_for_table` -- check read/write query activity
- - `get_warehouses` -- list available warehouses
+ - `analyze_storage_costs` -- runs the full analysis pipeline and returns pre-formatted output
+ - `get_asset_lineage` -- used only for follow-up lineage checks
+ The `analyze_storage_costs` tool supports **Snowflake, BigQuery, Redshift, and Databricks** warehouses only. Other warehouse types are out of scope.
+
## Workflow
- ### Step 1: Identify the scope
+ **Important:** These steps are internal instructions for you. Do NOT expose step numbers, step names, or the procedural structure to the user. Just act naturally.
- Ask the user which warehouse to analyze, or use the one they mentioned. Call `get_warehouses` to list available warehouses if needed.
+ ### Step 1: Identify the warehouse
- If the user specifies a schema or tag filter, use `search` with the appropriate filters to scope the analysis.
+ You need a warehouse to proceed.
- ### Step 2: Find candidate tables
+ - **If the user specified a warehouse** (by name or UUID), use it.
+ - **If not:** call `analyze_storage_costs` with no `warehouse_id`. The tool will either auto-pick when only one supported warehouse exists, or return a list of supported warehouses — let the user choose one, then call the tool again with the chosen `warehouse_id`.
- Use `search` to find tables that may be waste candidates. Run multiple searches to cover different patterns:
+ ### Step 2: Run the analysis
- 1. **Unmonitored tables**: `search(query="*", is_monitored=false, resource_ids=[warehouse_id])` -- tables nobody cared enough to monitor
- 2. **All tables** (if the user wants a full analysis): paginate through `search` results for the target warehouse
+ Call `analyze_storage_costs` with:
- For each candidate, note: table name, importance score, monitoring status, MCON.
+ - `warehouse_id`: the warehouse UUID
- ### Step 3: Investigate each candidate
+ The tool fetches candidates, classifies them into waste patterns (Unread, Write-only, Dead-end, Static waste, Zombie, Other stale) and table categories (Temporary, Archive/Snapshot, Production, Other), computes safety tiers, and returns a formatted analysis.
- For each candidate table (or top N by size), gather evidence:
+ - If the tool returns an error, report it to the user and stop.
+ - If no candidates are found, tell the user and stop.
- 1. **Query activity**: Call `get_queries_for_table(mcon=table_mcon, query_type="source")` to check reads, and `query_type="destination"` to check writes. Focus on:
- - Total read count (zero reads = potential waste)
- - Last read timestamp (stale if >90 days ago)
- - Write frequency (write-only tables are waste candidates)
+ ### Step 3: Present the initial summary
- 2. **Downstream dependencies**: Call `get_asset_lineage(mcons=[table_mcon], direction="DOWNSTREAM")` to check if anything consumes this table.
- - `has_relationships: false` = no downstream consumers (safer to remove)
- - Has downstream consumers = **do NOT recommend removal** without user review
+ The tool output contains two regions:
- 3. **Table metadata**: Call `get_table(mcon=table_mcon)` for size, type, and last update time.
+ 1. A `<!-- PRESENT_AS_IS -->` block with a condensed summary, a Top-N table, and a drill-down prompt.
+ 2. A `<!-- CATEGORY_DETAILS -->` block with per-category tables wrapped in `<!-- CATEGORY:<key> -->` markers. Do NOT present these yet.
- ### Step 4: Classify waste patterns
+ Present ONLY the `<!-- PRESENT_AS_IS -->` block — copy it verbatim, preserving every column, row, and value. Add a brief intro sentence if needed, then paste the block unchanged. The user will see the summary and top tables, then choose a category to drill into.
- Read the `references/waste-patterns.md` file and classify each table into one of the waste categories based on the evidence gathered. Apply the safety tier computation from `references/safety-tiers.md`.
+ **CRITICAL — do NOT call any other tool after `analyze_storage_costs` succeeds.** No `search`, no `get_table`, no troubleshooting agents, no cross-checks. The analysis result IS the final answer; your only remaining job is to present the `<!-- PRESENT_AS_IS -->` block verbatim.
- ### Step 5: Present recommendations
+ **CRITICAL — preserve markdown-linked MCONs verbatim.** The pre-formatted tables already contain properly linked MCONs (e.g., `` [`db:schema.table`](https://getmontecarlo.com/assets/MCON++...) ``). Never output bare MCON strings as plain text.
- Group findings by waste pattern and present to the user:
+ ### Step 4: Handle follow-up requests
- 1. **Safe to remove** (safety tier 0-1): Tables with no downstream dependencies, no reads, low importance. Recommend `DROP TABLE` or archival.
- 2. **Needs review** (safety tier 2-3): Tables with some risk factors. Present the evidence and let the user decide.
- 3. **Needs lineage investigation** (has downstream deps): Tables that have consumers -- **never recommend removal** without the user verifying downstream impact.
+ **Category drill-downs.** When the user asks about a specific category ("show me temporary tables", "what about production?", "tell me more about archive"):
- For each recommendation, include:
- - Table name (human-readable, never MCONs)
- - Waste pattern (e.g., "Unread -- zero queries in 90 days")
- - Size and estimated monthly cost (Snowflake: ~$23/TB/month)
- - Safety tier with explanation
- - Specific action: DROP, ARCHIVE, INVESTIGATE, or KEEP
+ 1. Find the matching `<!-- CATEGORY:<key> -->` section in the `analyze_storage_costs` result already in the conversation. **Do NOT re-invoke `analyze_storage_costs`** — the data is already there.
+ 2. Present that section's content verbatim — every column, row, and value.
+ 3. After presenting, remind the user of remaining categories they haven't explored yet.
- ### Important rules
+ Category keywords (see `references/output-structure.md` for the full list):
- - **Never recommend removing a table with downstream consumers** without explicit lineage verification. Safety first.
- - **Always explain WHY** a table is waste -- don't just say "drop it."
- - **Cost estimates are approximate.** Snowflake: ~$23/TB/month. For non-Snowflake warehouses, show size only (no cost estimate) and note that pricing varies.
- - **Importance scores are computed metrics**, not business criticality. A low importance score doesn't mean the table is safe to remove -- always check lineage and query activity.
- - **Present results as a table** for easy scanning: table name, waste pattern, size, safety tier, recommendation.
- - **Never expose MCONs, UUIDs, or internal identifiers** to the user.
+ - "temporary", "staging", "tmp", "stg" → `CATEGORY:temporary`
+ - "archive", "snapshot", "backup", "old" → `CATEGORY:archive_snapshot`
+ - "uncategorized", "other", "unknown" → `CATEGORY:other`
+ - "production", "prod", "critical", "important" → `CATEGORY:production`
+
+ If the user says "show me everything" or "all categories", present all category sections in order: temporary → archive → uncategorized → production.
+
+ **Lineage checks.** When the user asks what consumes a specific table ("check lineage for X", "is it safe to remove Y?", "what depends on this table?"):
+
+ 1. Call `get_asset_lineage` with `mcons: [<table mcon>]` and `direction: "DOWNSTREAM"`.
+ 2. If `has_relationships: false` → the table's consumers are likely BI dashboards or tools (not other tables). Mention this — it may still be safe to remove, but the user should verify with dashboard owners.
+ 3. If downstream tables exist AND are also stale → recommend removing both.
+ 4. If downstream tables are active → flag as risky, do NOT recommend removal.
+
+ **Note:** The `N consumers` flag in the Usage & Risk column counts ALL consumers, including BI dashboards (Looker, Tableau, Power BI) and other non-table assets. The lineage tool only returns table-to-table edges, so lineage results may show fewer consumers than the count. When that happens, explain the gap to the user.
+
+ ## Reading the Usage & Risk column
+
+ Each row's final `Usage & Risk` cell combines read-side activity with risk flags. Format:
+
+ ```
+ {activity} # no flags fire
+ {activity}; {flag1, flag2, ...} # one or more flags fire
+ ```
+
+ **Activity values** (always present):
+
+ - `No reads` -- no recorded reads
+ - `180d · 0 reads` -- last read N days ago, zero total reads
+ - `2d · 580 reads / 14 users` -- recent reads, total reads and distinct reading users
+
+ A low `days since read` is only meaningful when paired with the read count — a single backup job or security scanner can make a cold table look "1d". Always weigh staleness against reads + users.
+
+ **Risk flags** (appended after `; ` in this fixed order when any fire):
+
+ - `high criticality` / `medium criticality` -- pre-computed criticality
+ - `N consumers` -- has active consumers (tables, views, or BI dashboards); verify before removing
+ - `high importance score` -- `is_important` is a thresholded `importance_score ≥ 0.6` computed upstream in Databricks, **not** a user-applied tag
+ - `has monitors` -- actively monitored by Monte Carlo
+
+ ## Table categories
+
+ Tables are automatically classified for prioritized review:
+
+ - **Temporary/Staging** -- Short-lived ETL/test tables (safest to drop)
+ - **Archive/Snapshot** -- Historical copies, date-suffixed tables (verify retention policies)
+ - **Production** -- Monitored, critical, or lineage-important tables (highest risk)
+ - **Other** -- No strong signal either way (needs manual review)
+
+ ## Scope limitations
+
+ - **Storage** costs only -- not compute, query optimization, or billing
+ - One warehouse per analysis
+ - **Snowflake, BigQuery, Redshift, and Databricks** only
+ - **Recommendations only** -- never execute DROP TABLE or destructive actions