data-contract-extractor · git:20260813.83125c6 · 2026-08-13 · sha256 c0e97a0ac5cfc0d4
data-contract-extractor git:20260813.83125c6A
Immutable. This exact content never changes and is served at /api/v1/blob/c0e97a0ac5cfc0d4.
---
name: data-contract-extractor
description: "Reads a data-product spec (data PRD, dataset README, lineage doc) and emits a structured data contract - schema (columns + types + nullability + PII flags), freshness SLA, volume bounds, distribution invariants, and ownership. The contract is consumable by data-quality tools such as dbt tests, Great Expectations, or Soda checks as their assertion baseline. Use when scoping a new data product or formalizing assertions on an existing one."
---
# data-contract-extractor
## Overview
This skill formalizes the prose of a data PRD into a **data contract** -
the producer/consumer agreement on schema, freshness, volume, semantic
invariants, and ownership - that the QA test suite can mechanically
assert against, typically via the sibling skills in this plugin
(`dbt-testing`, `great-expectations`, `soda-checks`). "Data contract" is
practitioner-emergent terminology (Andrew Jones / Chad Sanderson), not
ISTQB-canonical; this skill cites industry-engineering sources.
## When to use
- A new data product (dbt model, table, file feed) is being scoped
and the team wants assertions before shipping.
- An existing data product has implicit assumptions consumers rely
on; the team is documenting them ahead of a refactor.
- A PRD or design doc references a dataset and the team needs the
schema / SLA pulled out into structured form.
- A data-quality engineer needs the contract as input to suite generation.
## What goes in a data contract
The five required sections, in order of authoring priority:
### 1. Schema
For each column:
| Field | Required | Notes |
|---------------|----------|-------|
| Name | yes | snake_case; matches the warehouse table. |
| Type | yes | Warehouse-native type (`VARCHAR`, `BIGINT`, `TIMESTAMP`, etc.). |
| Nullable | yes | `true` / `false` - the test suite's `not_null` decision. |
| PK | yes | `true` for primary-key column(s). |
| Unique | yes | `true` for candidate keys (separate from PK). |
| FK | optional | If foreign key, the `<table>.<column>` reference. |
| PII | required | Tag values: `none`, `direct` (email/name), `indirect` (zip, dob alone), `sensitive` (SSN, payment). |
| Constraint | optional | Range / enum / regex (per the sibling `data-quality-conventions` skill). |
| Description | yes | One-sentence semantic meaning; not just the type. |
### 2. Freshness SLA
| Field | Notes |
|-------------------|-------|
| Update cadence | `daily / hourly / continuous / weekly`. |
| Max staleness | The point past which downstream consumers should treat the data as broken (per the sibling `data-quality-conventions` skill: typically 2× cadence). |
| Source-of-truth column | The timestamp column the freshness check reads (e.g. `updated_at`, `loaded_at`, `event_time`). |
### 3. Volume bounds
| Field | Notes |
|-------------------|-------|
| Expected min/max | Row-count range under normal operation. |
| Volatility | `stable / monotonic-growth / cyclical / event-driven`. |
| Recovery action | If volume falls outside bounds, what does the consumer do? |
### 4. Distribution invariants
For each business-meaningful column, what must hold:
| Field | Example |
|------------------------------|----------------------------------------------------|
| Categorical: accepted values | `status ∈ {placed, shipped, completed, returned}`. |
| Numeric: range | `discount_pct ∈ [0, 100]`. |
| Frequency / rate | `cancellation_rate ≤ 5%` (rolling 7-day window). |
### 5. Ownership and governance
| Field | Notes |
|--------------|-------|
| Producer team| Who runs the pipeline. |
| Owner handle | Routing handle (Slack / email) for breaks. |
| Consumers | Known downstream models / dashboards / services. |
| Versioning | How breaking changes are communicated. |
## Output format
Emit as YAML for direct consumption by dbt / GX / Soda:
```yaml
# data-contracts/<dataset-slug>.yml
contract_version: 1
dataset:
name: orders
description: One row per customer order; updated within 1h of placement.
schema:
- name: order_id
type: BIGINT
nullable: false
pk: true
unique: true
pii: none
description: Surrogate key.
- name: customer_id
type: BIGINT
nullable: false
fk: customers.customer_id
pii: indirect
description: Customer who placed the order.
- name: email
type: VARCHAR
nullable: false
pii: direct
constraint: { format: email }
description: Customer email at time of order (immutable snapshot).
- name: status
type: VARCHAR
nullable: false
constraint:
accepted_values: [placed, shipped, completed, returned]
description: Current fulfillment status.
- name: discount_pct
type: DECIMAL(5,2)
nullable: true
constraint: { range: [0, 100] }
description: Promotion discount applied; null = no promo.
- name: updated_at
type: TIMESTAMP
nullable: false
description: Last-modified timestamp; freshness source-of-truth.
freshness:
cadence: hourly
max_staleness: 2h
source_column: updated_at
volume:
min_per_day: 100
max_per_day: 1000000
volatility: cyclical
recovery_action: |
Investigate ingestion pipeline at <runbook-url> if volume falls outside bounds for >2 cycles.
distribution:
- column: status
rule: accepted_values
values: [placed, shipped, completed, returned]
- column: discount_pct
rule: range
range: [0, 100]
- column: cancellation_rate
rule: rolling_window
window: 7d
max: 0.05
ownership:
producer_team: data-platform
owner: '@data-platform-oncall'
consumers:
- dbt: marts/orders_summary
- dashboard: 'Order Health'
- service: shipping-microservice
versioning: |
Breaking changes coordinated via #data-contracts Slack 14d in advance.
```
## Step-by-step extraction
When reading a data PRD:
1. **Schema** - look for table-spec sections, ER diagrams, or
per-column descriptions. If only column names are given, flag
missing types as gaps; do not guess.
2. **PII tagging** - every column gets a `pii:` tag, even `pii: none`.
Force the data-product author to confirm; PII handling drives
downstream architecture.
3. **Freshness** - look for "updated daily" / "real-time" / "<X
minutes." If absent, flag as a gap.
4. **Volume** - look for traffic projections or current scale. If
absent, flag as a gap to be filled before the contract is
actionable.
5. **Distribution** - look for business rules in prose form ("status
is one of...", "discount up to 100%") and translate.
6. **Ownership** - look for the responsible team / Slack channel.
## Gap flagging
The agent never fabricates contract fields. Every gap becomes an
explicit question:
```markdown
## Contract gaps (HUMAN INPUT REQUIRED)
| Section | Field | Question |
|---------------|------------------------|--------------------------------------------------------------------------|
| schema | `customer_phone.pii` | Direct PII (yes - phone numbers are PII per most jurisdictions). Confirm. |
| freshness | `cadence` | PRD says "fresh data" - daily? hourly? real-time? Each implies a different gate. |
| volume | `max_per_day` | Not specified; required to set a `row_count between` assertion. |
| distribution | `payment_method` | What's the accepted-value set? PRD lists four; are there more (e.g. "apple_pay" added recently)? |
```
Until the gaps are filled, the contract is incomplete and the test
suite cannot be generated.
## Examples
Three worked examples (PRD to contract, minimal source with a gap list,
and refactor of an existing dataset) are in
[references/examples.md](references/examples.md).
## Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---------------------------------------------------|---------------------------------------------------------------|-----|
| Auto-tagging every column `pii: none` | Misses real PII; downstream ungoverned. | Force the author to make the call; default to "needs review" rather than "none". |
| Generic freshness "real-time" | Ambiguous; "real-time" varies from <100ms to <1min by team. | Quantify: `cadence: continuous, max_staleness: 60s`. |
| Distribution rules without a window | "Cancellation rate ≤5%" - over what window? Lifetime? | Always specify a window: `rolling_window: 7d`. |
| Skipping Ownership | A contract without an owner is a wishlist; nobody's on the hook. | Require ownership before declaring the contract complete. |
## References
- ISTQB Glossary V4.7.1 - testability + non-functional testing
(cited in `non-functional-requirement-extractor`, in the qa-shift-left
plugin) - for the underlying observability heuristic.
- ISO/IEC 25012:2008 - data-quality model (cite by stable ID;
paywalled at iso.org).
- Sibling skills `dbt-testing`, `great-expectations`, `soda-checks` -
the downstream consumers of this skill's output.
- `data-quality-conventions` - naming and threshold conventions referenced from the contract.