agent-production-operations · diff

git:20260814.c0c7690 to git:20260814.a315b77

17 added, 17 removed. Audit A to A.

---

name: agent-production-operations

description: >-

Operate an evaluated agent with tools and authority in production through a

runtime control plane covering versioning, staged rollout, fallback, cost and

latency budgets, tool health, human escalation, disablement, and trace-to-eval

feedback. Do not use for building agents, designing evals, or general release,

SRE, security, or platform engineering — those methods stay with their

specialist skills.

license: MIT

compatibility: Agent harness with file read/write and terminal access. No network or runtime dependency required by the bundle itself.

metadata:

spec-version: "1.0"

tags: agent, production, operations, runtime, control-plane, rollout, fallback, disablement

---

# agent-production-operations

A runtime control plane for taking an evaluated agent with tools and authority

into controlled production operation. This bundle bridges the gap between

passing evaluations and safe production use: it defines how an agent is

versioned, rolled out, monitored, constrained, escalated, and disabled — and

how production evidence feeds back into evaluation and release decisions.

The bundle does **not** build agents, design evaluations, run release pipelines,

or own infrastructure. It composes those concerns from existing specialist

skills and owns only the runtime control decisions between them.

## When to load this

Load this bundle when you are operating an agent that:

- Has passed evaluation and is cleared for production consideration.

- Has tools with side effects and delegated authority that must be gated.

- Needs a staged rollout with progressive authority expansion.

- Must be monitored for cost, latency, tool health, and authority usage in

production.

- Requires a defined fallback, escalation, and disablement path.

- Should feed production traces and incidents back into evaluation cases and

release-gate updates.

## When not to use

Do **not** load this bundle for:

- Building or designing an agent from scratch — route to the appropriate

framework skill (LangGraph, CrewAI, AutoGen, etc.).

- Designing agent evaluations or observability — route to

- [agent-evals-and-observability](../../agent-evals-and-observability/SKILL.md).

+ [agent-evals-and-observability](../agent-evals-and-observability/SKILL.md).

- General release engineering (CI/CD pipelines, artifact promotion) — route to

- [release-engineering](../../release-engineering/SKILL.md).

+ [release-engineering](../release-engineering/SKILL.md).

- General site reliability engineering (incident response, SLO definition,

error budgets) — route to

- [site-reliability-engineering](../../site-reliability-engineering/SKILL.md).

+ [site-reliability-engineering](../site-reliability-engineering/SKILL.md).

- General security engineering (threat modeling, vulnerability assessment) —

- route to [secure-software-engineering](../../secure-software-engineering/SKILL.md).

+ route to [secure-software-engineering](../secure-software-engineering/SKILL.md).

- General platform infrastructure (compute, networking, storage) — route to

- [platform-engineering](../../platform-engineering/SKILL.md).

+ [platform-engineering](../platform-engineering/SKILL.md).

- Performing a production-readiness review — route to

- [production-readiness](../../production-readiness/SKILL.md); this bundle

+ [production-readiness](../production-readiness/SKILL.md); this bundle

**consumes** readiness outcomes, it does not produce them.

- Conducting an incident postmortem — route to

- [incident-learning](../../incident-learning/SKILL.md); this bundle

+ [incident-learning](../incident-learning/SKILL.md); this bundle

**consumes** incident records as escalation and feedback inputs.

## Autonomy is not assumed uniform

This bundle explicitly does **not** assume all agents share the same autonomy,

user population, or side-effect profile. Agent profiles are distinguished by

capability class and side-effect surface:

|---|---|---|---|---|

| **Side-effect-capable, customer-facing** | Support agent with account access | Read PII, suggest actions, no mutation without confirmation | Any PII access without explicit consent; cost breach > 10% of budget | Degrade to read-only, escalate immediately |

Every control-plan decision (authority, escalation, fallback, disablement) in

this bundle is parameterized by the agent's autonomy profile. A read-only agent

and a side-effect-capable agent operating on customer data receive different

thresholds and different escalation paths.

## Runtime control plane routing

The bundle composes the following specialist skills. Load them when their

domain is the active concern; the bundle owns the cross-domain coordination.

|---|---|---|---|

- | Agent evaluation and observability | [agent-evals-and-observability](../../agent-evals-and-observability/SKILL.md) | Designing evals, instrumenting observability, detecting regression | Trace-to-eval feedback loop; eval-case generation from production incidents |

- | Release engineering | [release-engineering](../../release-engineering/SKILL.md) | Release pipeline design, artifact promotion, CI/CD | Agent-specific staged rollout with authority gating; rollback triggers tied to agent health |

- | Site reliability engineering | [site-reliability-engineering](../../site-reliability-engineering/SKILL.md) | Incident response, SLO/SLI definition, error budgets | Agent-specific latency/cost budgets; tool-health SLOs; escalation handoff to incident command |

- | Security engineering | [secure-software-engineering](../../secure-software-engineering/SKILL.md) | Threat modeling, trust-boundary validation, secure design | Agent authority contracts with side-effect boundaries; disablement security (revoke, not just stop) |

- | Incident learning | [incident-learning](../../incident-learning/SKILL.md) | Post-incident analysis, verified closure, follow-up work | Consumed as input: incident records feed escalation thresholds and eval-case generation |

+ | Agent evaluation and observability | [agent-evals-and-observability](../agent-evals-and-observability/SKILL.md) | Designing evals, instrumenting observability, detecting regression | Trace-to-eval feedback loop; eval-case generation from production incidents |

+ | Release engineering | [release-engineering](../release-engineering/SKILL.md) | Release pipeline design, artifact promotion, CI/CD | Agent-specific staged rollout with authority gating; rollback triggers tied to agent health |

+ | Site reliability engineering | [site-reliability-engineering](../site-reliability-engineering/SKILL.md) | Incident response, SLO/SLI definition, error budgets | Agent-specific latency/cost budgets; tool-health SLOs; escalation handoff to incident command |

+ | Security engineering | [secure-software-engineering](../secure-software-engineering/SKILL.md) | Threat modeling, trust-boundary validation, secure design | Agent authority contracts with side-effect boundaries; disablement security (revoke, not just stop) |

+ | Incident learning | [incident-learning](../incident-learning/SKILL.md) | Post-incident analysis, verified closure, follow-up work | Consumed as input: incident records feed escalation thresholds and eval-case generation |

## Loading protocol

This SKILL.md is the discoverable umbrella entry point. Nested skills are not

used in this bundle. Reference files are loaded on trigger:

| Reference | Loaded when |

|---|---|

| [references/discovery-brief.md](references/discovery-brief.md) | Reviewing the bundle's boundary decisions against specialist skills |

| [references/agent-production-contract.md](references/agent-production-contract.md) | Defining capability, authority, uncertainty, escalation, and side-effect contracts for an agent |

| [references/runtime-control-plan.md](references/runtime-control-plan.md) | Planning versioning, staged rollout, or fallback for an agent in production |

| [references/tool-authority-health.md](references/tool-authority-health.md) | Recording or reviewing tool availability, failure, and authority usage/breach state over time |

| [references/trace-to-eval-feedback.md](references/trace-to-eval-feedback.md) | Connecting production traces and incidents back to evaluation cases and release gates |

## Concrete operational parameters

### Authority

- **Definition**: the set of actions an agent is permitted to perform, scoped by

target (which resources), operation (read/write/delete), and user-context

(whose data).

- **Trigger to review**: any new tool registration, model update, or prompt

change that expands the agent's reachable action surface.

- **Threshold**: authority is binary per action class. No action may be

performed that is not explicitly listed in the production contract. An

attempt to perform an unauthorized action is an **authority breach** and

triggers immediate escalation.

- **Action on breach**: log the attempt, block the action, increment the

authority-breach counter, and escalate. If breach count exceeds 3 in a

rolling 24-hour window, disable the agent.

### Escalation

- **Definition**: transfer of a decision or action from the agent to a

designated human operator.

- **Triggers** (any one triggers escalation):

- Authority breach (attempted unauthorized action).

- Uncertainty above threshold: agent confidence < 0.7 on a

side-effect-capable action (configurable per profile).

- Tool failure rate > 5% in a 5-minute sliding window.

- Cost budget breach > 10% of allocated budget in a billing period.

- Latency p95 > 2x baseline for > 5 minutes.

- Human-handoff keyword or explicit user escalation request.

- **Action**: suspend the agent's side-effect authority, log the escalation

context (trigger, state snapshot, pending actions), notify the designated

escalation channel, and await human disposition (resume / reduced-authority /

disable).

### Fallback

- **Definition**: a predetermined safe behavior when the agent, model, or a

tool cannot operate at its normal capability level.

- **Triggers**:

- Model endpoint returns 5xx for > 30 seconds.

- A critical tool is unavailable (health-check failure for > 2 minutes).

- Cost budget exhausted (100% consumed).

- Latency p95 > 5x baseline for > 2 minutes.

- **Actions per profile**:

- Read-only: return a static/cached response with a "results may be stale"

disclaimer.

- Side-effect-capable, internal: degrade to read-only, queue pending

mutations, notify operator.

- Side-effect-capable, customer-facing: degrade to read-only, surface a

"temporarily unavailable" message to the user, escalate immediately.

### Disablement

- **Definition**: complete revocation of the agent's ability to act, including

read-only access. Distinct from fallback (which preserves reduced capability).

- **Triggers**:

- 3 or more authority breaches in a 24-hour window.

- Production-readiness review outcome is "no-go" or "defer" for the current

agent version.

- Incident-learning record attributes a severity-1 or severity-2 incident to

agent action, and verified closure is not yet complete.

- Human operator issues an explicit disable directive.

- Tool compromise or credential leak detected (route through

- [secure-software-engineering](../../secure-software-engineering/SKILL.md)).

+ [secure-software-engineering](../secure-software-engineering/SKILL.md)).

- **Action**: revoke all credentials and tokens; remove from routing/load-balancing;

record disablement reason, timestamp, and authorizing evidence; notify

escalation channel; prevent automatic restart until a new

production-readiness review passes.

### Cost

- **Definition**: the financial cost of operating the agent, attributed to a

budget owner.

- **Budget**: allocated per agent per billing period (e.g., $500/day for a

customer-facing support agent). Budget is set at production-contract time and

reviewed at each production-readiness cycle.

- **Thresholds**:

- 50% consumed: notification to budget owner.

- 80% consumed: warning; cost-optimization review triggered.

- 100% consumed: fallback to degraded mode (read-only or cached).

- 110% consumed: disablement (hard stop, no further cost accrual).

- **Measurement**: per-request model cost + per-call tool cost (where

applicable), attributed to the agent instance. Cost data is appended to the

tool-authority-health record.

### Latency

- **Definition**: end-to-end response time from user request to agent response,

measured at p50 and p95 over 5-minute windows.

- **Baseline**: established during staged rollout observation window (phase 1

of rollout). Recorded in the agent production contract.

- **Thresholds**:

- p95 > 2x baseline for > 5 minutes: escalation trigger.

- p95 > 5x baseline for > 2 minutes: fallback trigger.

- p50 > 3x baseline for > 10 minutes while cost budget is > 80% consumed:

disablement consideration.

- **Action**: latency breaches feed the trace-to-eval feedback loop: the

latency-impacted requests are sampled and reviewed for eval-case generation.

### Privacy

- **Definition**: constraints on agent access to, processing of, and retention

of user data (PII, usage patterns, conversation content).

- **Concrete rules**:

- An agent must not access PII without an explicit user-data-access grant in

its authority contract.

- Agent traces and conversation logs must be scrubbed of PII before entering

the trace-to-eval feedback pipeline (route scrubbing design to

- [privacy-engineering](../../privacy-engineering/SKILL.md)).

+ [privacy-engineering](../privacy-engineering/SKILL.md)).

- Data retention for agent traces defaults to 30 days unless a shorter period

is specified in the production contract.

- Any PII access by a read-only agent triggers immediate escalation (the

read-only profile should never touch PII).

- **Privacy breach**: any PII access outside the granted scope triggers

escalation and a mandatory security review via

- [secure-software-engineering](../../secure-software-engineering/SKILL.md).

+ [secure-software-engineering](../secure-software-engineering/SKILL.md).

## Core workflow

1. **Contract**: define the agent's capability, authority, uncertainty,

escalation, and side-effect contracts using

[references/agent-production-contract.md](references/agent-production-contract.md).

Consume production-readiness outcomes and incident-learning records as

decision inputs.

2. **Version and plan**: version the model, prompt, tools, policy, and

evaluator; plan the staged rollout and fallback path using

[references/runtime-control-plan.md](references/runtime-control-plan.md).

3. **Roll out**: execute staged rollout with progressive authority expansion,

monitoring tool health and authority usage at each stage using

[references/tool-authority-health.md](references/tool-authority-health.md).

4. **Monitor**: track cost, latency, tool health, authority breaches, and

escalation events. Apply threshold-based actions (notify, degrade, escalate,

disable) per the concrete parameters above.

5. **Feed back**: connect production traces and incidents to evaluation cases

and release-gate updates using

[references/trace-to-eval-feedback.md](references/trace-to-eval-feedback.md).

6. **Learn**: consume incident-learning verified-closure records and

production-readiness review outcomes to update agent authority, escalation

thresholds, and disablement conditions.

## File map

| Path | Purpose |

|---|---|

| [SKILL.md](SKILL.md) | Umbrella entry point (this file) |

| [README.md](README.md) | Human-facing overview and quick start |

| [AGENTS.md](AGENTS.md) | Agent loading and discovery notes |

| [references/discovery-brief.md](references/discovery-brief.md) | Boundary analysis against specialist skills |

| [references/agent-production-contract.md](references/agent-production-contract.md) | Capability, authority, uncertainty, escalation, and side-effect contracts |

| [references/runtime-control-plan.md](references/runtime-control-plan.md) | Versioning, staged rollout, and fallback plan |

| [references/tool-authority-health.md](references/tool-authority-health.md) | Tool availability/failure and authority usage/breach record |

| [references/trace-to-eval-feedback.md](references/trace-to-eval-feedback.md) | Production-to-evaluation feedback loop |

| [evals/evals.json](evals/evals.json) | Integrated output-quality evaluation cases |

| [manifest.yaml](manifest.yaml) | Machine-readable composition contract (schema v1): purpose, audience, stages, included skills, prerequisites, outputs, handoffs, conflicts, and eval suite; consumed by the lifecycle capability matrix |