v1.0.0 to v1.0.0

142 added, 59 removed. Audit A to A.

---

name: aws-vpc-dns-investigation

description: Use this skill when a name is not resolving as expected inside a VPC, or before applying a DNS control-plane change. Activate on symptoms such as NXDOMAIN or SERVFAIL from an EC2 instance, a hostname resolving to a public address when a private endpoint was expected, an AWS service endpoint that stopped resolving after a VPC endpoint or Route 53 change, an application reaching the wrong IP, resolution that works from one instance but not another, IPv6 or dualstack resolution differences, a suspected on-premises forwarding or hybrid DNS problem, or a request to check whether enabling private DNS, adding a Resolver rule, associating a private hosted zone, attaching DNS Firewall, or associating a Route 53 Profile would break anything. It drives the aws-vpc-dns-diagnostics MCP server to observe live resolution from inside the subnet and to simulate a proposed change before it is applied.

metadata:

author: ddericco

version: "1.0.0"

aws-devops-agent-skills.agent-types: "Chat tasks, Incident RCA"

aws-devops-agent-skills.aws-services: "Amazon VPC, Amazon Route 53, Amazon EC2, AWS Systems Manager"

aws-devops-agent-skills.technical-domains: "Networking"

---

# Investigate VPC DNS Resolution

- Use the tools on the connected `aws-vpc-dns-diagnostics` MCP server. Start by calling

- `list_sops`, then `get_sop` with slug `Z-general-triage` to load the triage decision

- tree, and follow the runbook it returns. The runbooks are the authoritative

- procedure; this skill decides when to engage and in what order.

+ Use the tools on the connected `aws-vpc-dns-diagnostics` MCP server.

- ## Establish preconditions before interpreting any result

+ ## Step 1: Classify the request as Mode A or Mode B

- Call `dns_probe_context` first. A resolution result means nothing until you know

- whether the VPC resolver is even answering.

+ Before calling any tool, determine which mode applies:

- `enableDnsSupport` gates the entire VPC resolver. When it is false, neither the

- `.2` address nor the IPv6 resolver answers at all, and every downstream symptom is

- explained by that one attribute. The same call returns the instance's address

- family and the VPC DHCP option set's `domain-name-servers`, which is the resolver

- the VPC intends the instance to use. Load `A-resolver-disabled-precondition` when

- the attribute is false.

+ - **Mode A (live diagnosis):** The operator reports a resolution symptom from a

+ running instance. They provide an instance ID (or you can identify one). The

+ goal is to observe what actually resolves and compare resolvers.

+ - **Mode B (pre-change validation):** The operator asks whether a proposed DNS

+ change is safe. They provide account, region, VPC, and a change descriptor.

+ No instance is required.

- ## Observe what actually resolves

+ If the request is ambiguous, ask the operator to clarify. Do not default to

+ Mode A when the input lacks an instance ID, and do not default to Mode B when

+ the operator describes a live symptom.

- For a live symptom, call `dns_probe_compare` with the failing name. It runs an

- allowlisted probe set inside the instance through SSM and returns each resolver's

- answer alongside the resolver's own identity from `hostname.bind`, so you learn

- which resolver answered rather than assuming. The VPC DHCP resolver is added

- automatically, so a custom or hybrid resolver is compared against the VPC resolver

- without you looking it up first.

+ ## Step 2: Load safety rules

- Compare the instance's actual `/etc/resolv.conf` against the DHCP option set from

- `dns_probe_context`. A mismatch means the instance is not using the resolver the

- VPC hands out, which is a different root cause from a misconfigured rule.

+ Regardless of mode, call `get_sop` with slug `A-critical-safety-rules` and

+ follow every rule it contains. These are non-negotiable constraints on how you

+ interpret results, handle opaque constructs, and report findings.

- Judge answers by name category, not by whether resolvers agree. Two resolvers

- returning the same wrong answer is still a failure, and a divergence can be

- correct. Load `A-name-category-classification` before concluding.

+ ---

- ## Validate a change before it is applied

+ ## Mode A route: live diagnosis

- When the request is whether a change is safe, call `dns_simulate_effective_config`

- to get the VPC's effective configuration, which is the union of directly attached

- resources and anything inherited through an associated Route 53 Profile, with each

- construct tagged by its source. Then call `dns_simulate_change` with the proposed

- change to get a per-name impact report. This is symbolic and read-only; it predicts

- breakage without touching the control plane.

+ ### Required inputs

- Never recommend applying one of these changes without simulating it first. A broad

- FORWARD rule, enabling private DNS on an interface endpoint, or a Profile

- association can silently redirect names that currently resolve correctly.

+ account_id, region, instance_id, and the failing DNS name.

- ## Load the matching pattern runbook

+ ### Tool sequence (in order)

- When a signature appears in the output, retrieve the runbook for it with `get_sop`

- rather than reasoning from first principles. Available patterns include custom or

- hybrid resolver divergence, FORWARD versus private hosted zone precedence

- collisions, address-family divergence, VPC endpoint shadow NXDOMAIN, broad FORWARD

- sweep, DNS Firewall blocks, the `privateDnsEnabled` and `PrivateDnsPreference`

- flag-AND mismatch, and Route 53 Profile propagation timing. Call `list_sops` for the

- current catalogue and exact slugs.

+ 1. `dns_probe_context` — establishes VPC-attribute preconditions: enableDnsSupport,

+ enableDnsHostnames, address family, DHCP option set. A resolution result means

+ nothing until you know whether the VPC resolver is answering.

+ 2. `dns_probe_compare` — runs the allowlisted probe set inside the instance via

+ SSM. Returns each resolver's answer and the resolver's own identity from

+ `hostname.bind`. The VPC DHCP resolver is auto-added for comparison.

+ 3. `get_sop` — load the pattern runbook matching the observed signature

+ (see trap-to-SOP mapping below).

- ## Report honestly

+ ### Interpretation rules

- All tools are read-only observation and simulation. Do not modify, delete, or

- create DNS resources as part of this skill; produce the diagnosis and the

- recommended change, and leave application to the operator.

+ - If `enableDnsSupport` is false: load `A-resolver-disabled-precondition`. The

+ VPC resolver is intentionally dark and every probe failure follows from that.

+ - Compare the instance's `/etc/resolv.conf` (from the probe output) against the

+ DHCP option set. A mismatch means the instance is not using the VPC-intended

+ resolver.

+ - Judge answers by name category (load `A-name-category-classification`), not by

+ whether resolvers agree. Two resolvers returning the same wrong answer is still

+ a failure.

- Cross-account constructs shared with the target account may be enumerable but

- opaque, and the tools mark them as such. Report an opaque construct as unknown

- rather than treating it as absent. Load `C-cross-account-opaque-constructs` and

- `C-limitations-and-boundaries` and state the boundaries to the operator instead of

- inferring past them.

+ ### Mode A trap-to-SOP mapping

- Requires the aws-vpc-dns-diagnostics MCP server to be registered in the Agent Space

- with its tools allowlisted. The server is in this repository at

- `mcp/aws-vpc-dns-diagnostics-mcp/`. Mode A tools additionally require the target

- instance to be reachable through SSM. If the server is not registered or SSM is

- unreachable, report that as the blocker rather than guessing at the resolution

- path.

+ | Observed signature | SOP slug |

+ | --- | --- |

+ | Custom resolver answers differently from VPC .2 | `A-custom-resolver-divergence` |

+ | FORWARD rule and PHZ both match the name | `A-forward-vs-phz-precedence-collision` |

+ | A record works, AAAA fails (or vice versa) | `A-address-family-divergence` |

+ | enableDnsSupport is false | `A-resolver-disabled-precondition` |

+ | General live comparison procedure | `A-mode-a-live-resolver-comparison` |

+ ### Reporting format for Mode A

+ Label every finding as **Observed** (ground truth from the probe). State which

+ resolver answered and what it returned. When Mode A and Mode B produce different

+ conclusions for the same name, **Mode A wins** because it is ground truth from

+ inside the subnet.

+ ---

+ ## Mode B route: pre-change validation

+ ### Required inputs

+ account_id, region, vpc_id, and a change descriptor (structured dict with `type`

+ and type-specific fields). No instance required.

+ ### Tool sequence (in order)

+ 1. `dns_simulate_effective_config` — returns the VPC's effective DNS config: the

+ union of directly attached resources and anything inherited through an

+ associated Route 53 Profile, each construct tagged by source.

+ 2. `dns_simulate_change` — applies the proposed change symbolically and returns a

+ per-name impact report (before/after, delta, traps, severity, volume).

+ 3. `get_sop` — load runbooks for any traps reported in the impact table

+ (see trap-to-SOP mapping below).

+ ### Interpretation rules

+ - Never recommend applying a change without simulating it first. A broad FORWARD

+ rule, enabling private DNS on an interface endpoint, or a Profile association

+ can silently redirect names that currently resolve correctly.

+ - The candidate set is limited to API-derived names (PHZ records, rule domains,

+ VPCE apexes, Firewall domain lists) or operator-supplied names. It is not

+ exhaustive. State the coverage boundary.

+ - If the operator supplies `volumes` (from Resolver Query Logs), names are ranked

+ by traffic. This is enrichment; absence does not invalidate the simulation.

+ ### Mode B trap-to-SOP mapping

+ | Trap label in impact report | SOP slug |

+ | --- | --- |

+ | VPCE-shadow-NXDOMAIN | `B-vpce-shadow-nxdomain` |

+ | broad-FORWARD-sweep | `B-broad-forward-sweep` |

+ | flag-AND-mismatch | `B-flag-and-mismatch` |

+ | DNS-Firewall-block | `B-dns-firewall-block` |

+ | profile-union-shift | `B-profile-propagation-timing` |

+ | General pre-change procedure | `B-mode-b-pre-change-validation` |

+ ### Reporting format for Mode B

+ Label every finding as **Predicted** (symbolic, not ground truth). State the

+ candidate-set size, its source (API-derived or operator-supplied), and that names

+ outside this set were not evaluated. Include the propagation timing caveat for

+ Profile changes.

+ ---

+ ## Cross-account opacity

+ Call `get_sop` with slug `C-cross-account-opaque-constructs` when the effective

+ config or impact report contains opaque markers. Cross-account constructs shared

+ via RAM or a Route 53 Profile may be enumerable but their contents are not

+ readable from the consumer account. Report them as "present but unknown content"

+ rather than treating them as absent or inferring past them.

+ ## Limitations

+ Call `get_sop` with slug `C-limitations-and-boundaries` and state the relevant

+ boundaries to the operator. Key constraints:

+ - All tools are read-only. Do not modify, delete, or create DNS resources.

+ - Mode A requires SSM reachability (ssm, ssmmessages, ec2messages VPC endpoints

+ and an instance role with AmazonSSMManagedInstanceCore).

+ - Mode B candidate sets are not exhaustive. The "no impacts" conclusion applies

+ only within the tested set.

+ - Opaque constructs cannot be resolved from this account.

+ - Resolver Query Log ingestion is not implemented; volumes must be supplied by

+ the operator.

+ ## Final response requirements

+ Every response produced by this skill must include:

+ 1. Each finding labelled **Observed** (Mode A) or **Predicted** (Mode B).

+ 2. When both modes were used, state "Mode A wins" for any conflict.

+ 3. The candidate-set coverage: how many names, what source, what was not tested.

+ 4. Any opaque constructs and their impact on the conclusion.

+ 5. Recommended next steps or the specific change to apply (never apply it).

+ ## Prerequisites

+ Requires the aws-vpc-dns-diagnostics MCP server registered in the Agent Space

+ with its tools allowlisted. The server is at `mcp/aws-vpc-dns-diagnostics-mcp/`.

+ If the server is not registered or SSM is unreachable, report that as the blocker

+ rather than guessing at the resolution path.