cortex-platform-xdm-author v2.1.2A

Immutable. This exact content never changes and is served at /api/v1/blob/b768e4e6ef429683.

---
name: cortex-platform-xdm-author
description: Author Cortex XSIAM Data Model Rules in Cortex Query Language (XQL). Turns a raw vendor log sample into a production-ready rule with a MAPPED-header comment block.
version: 2.1.2
license: AGPL-3.0-or-later
---

# cortex-platform-xdm-author

Author Cortex XSIAM data model rules in Cortex Query Language (XQL) from raw vendor log samples.

The current version is 2.1.2; the per-version change history lives in [CHANGELOG.md](CHANGELOG.md). On top of base rule authoring, the skill provides these auto-detected mapping capabilities, each backed by an advisory linter check:

- Record-level classification and catch-all (new in 1.7.0) -- `xdm.event.type` and `xdm.event.tags` are decided per record via `if()`, tags come from the closed six-member `EVENT_TAG` enum, an unrecognised record falls through to blank tags, and nothing is filtered out (unclassified records get the `GOCORTEX_UNMODELLED` sentinel so the datamodel row count equals the raw count). WARN-045 flags an invented tag; WARN-046 flags a content filter that drops records without a catch-all. `if()` takes the FIRST true branch, so branch ORDER assigns the event: order most specific first, network last among the real kinds, and prefer CLASSIFY ONCE, DERIVE MANY -- one `tmp_class` decision that both the type and tags chains read -- because two parallel chains can disagree with both fields legal and the linter silent. See [references/record-classification.md](references/record-classification.md).
- Syslog Stage 0 envelope -- relay-aware host capture and priority decode for any syslog source (WARN-040 / WARN-041). HARD RULE, NO EXCEPTIONS: syslog reaches Cortex both direct off the device and behind an intermediate relay that prepends its own `<PRI>` header, and ONE rule must model BOTH in the same rule, ALWAYS -- never one form and never two rules. The envelope skips any relay prefix with a greedy `^.*` (origin host + origin PRI) and every payload field is anchored on its own token, never on `^` or a positional offset. A positional body capture is BLOCKED as ERR-030 (error, not advisory). Static lint cannot prove prepend-robustness on its own, so every syslog rule must also pass `python3 scripts/verify_rule.py <rule> <sample> --prepend-check`, which evaluates each record twice -- as supplied and relay-prepended -- and fails on any field whose value differs. See [references/syslog-envelope.md](references/syslog-envelope.md).
- Authentication Story -- the mandatory XDM field set for login / logon / MFA / SSO events, including the account-class fields `identity_type` / `user_type`, `xdm.auth.service` as the ROLE the system played in the flow, and `xdm.target.resource.name` as the entity the principal authenticated TO, with type-valid fail-safe padding (WARN-042). `xdm.auth.service` is `"IDP"` (this system validates the credential), `"SP"` (it initiates and relies on another to validate) or `"Universal"` (the source is not a known IdP provider), decided PER EVENT TYPE, since one feed routinely carries more than one role -- upstream Okta maps its SSO records to `"SP"` and its credential-validation records to `"IDP"` on the same dataset. It is NEVER a service name: until 1.9.0 this bundle taught the opposite, reasoning from the schema's plain `String` type to the absence of a vocabulary, which does not follow, since a String-typed field can still carry a documented closed vocabulary. The protocol or mechanism that was being written there belongs in `xdm.auth.auth_method` (TACACS+, RADIUS, LDAP), `xdm.network.application_protocol` (ssh, telnet, snmp) or `xdm.logon.package_name` (Kerberos, NTLM); an access channel (CLI, GUI, Dashboard, API) belongs in `xdm.target.application.name` and is barred from `xdm.event.operation_sub_type`, whose closed list it would corrupt. `"Universal"` is house law, registered in [references/house-conventions.md](references/house-conventions.md) rather than stated as schema fact. The target field is the one member of the set that must NEVER be padded (WARN-055): an authentication event has a direction, and until the target was mandatory nothing recorded it -- a shipped rule inverted 764 SSH logins by mapping the accessed router to `xdm.source.host.hostname`, with all fourteen then-mandatory fields assigned and the linter silent, and only a population count exposed it. A placeholder would satisfy the checker while leaving the event targetless, so the field is derived from the raw log or left to resolve null. Set it IN ADDITION to `xdm.target.host.hostname` / `xdm.target.application.name` / `xdm.target.ipv4`, never instead of them; this supersedes the cloud-only reading of `xdm.target.resource.*` for authentication events only, and is recorded as a house convention in [references/house-conventions.md](references/house-conventions.md). Windows logon (4624 / 4625) and Kerberos (4768 / 4769) are covered: the complete `LOGON_TYPE` list maps the `LogonType` integer, and the full Kerberos encryption-type / error-code enums ship as [assets/kerberos_crosswalk.json](assets/kerberos_crosswalk.json) (rendered by `scripts/kerberos_map.py`) -- see [references/authentication-mapping.md](references/authentication-mapping.md).
- Network / Firewall Story -- the mandatory XDM field set for firewall, flow, proxy, IDS/IPS and DNS traffic events (WARN-043). Network is the foundational layer, so an IDS or WAF event maps it on top of its primary role, and a dual event (a VPN login) takes both story sets. The three `xdm.network.http.*` leaves are NOT part of the unconditional set: they are mandatory only for a network event that carries an HTTP layer, which the rule declares by mapping another `xdm.network.http.*` field, a URL, or HTTP among its protocol layers. Never pad a header name, a header value and a URL category onto a router SSH login, an SNMP failure or a control-plane record -- that asserts a protocol the source never saw, and a requirement no honest router rule can satisfy only teaches authors to mute the checker.
- Endpoint telemetry -- process / file / registry / image events (Linux commands, Windows Security, Sysmon, EDR) on the channel/verb model: `xdm.event.type` = the channel/source, `xdm.event.original_event_type` = the per-event name, `xdm.event.operation` = the precise verb from the complete 56-member `OPERATION_TYPE` enum. Endpoint events carry blank `xdm.event.tags` (no process story tag). WARN-050 flags a process/file/registry event with no operation verb. See [references/process-mapping.md](references/process-mapping.md).
- Cloud audit story (AWS CloudTrail, Azure Activity / Entra sign-in, GCP Cloud Audit) -- the operation verb is DERIVED from the provider's action-naming convention (CloudTrail `eventName`, Azure `operationName`, GCP `methodName`), the outcome from `errorCode` / `resultType` / `status.code`, and the event is tagged CLOUD (plus AUTHENTICATION for a console login / sign-in). `xdm.*.cloud.provider` is mapped reliably; `xdm.*.cloud.service` (`CLOUD_SERVICE_TYPE`) is set only on a confident match, otherwise the raw service name is kept in `xdm.event.type` / `xdm.event.description` or the NOT MAPPED block -- never `xdm.*.cloud.source_type`, which is a banned internal-only XCloud asset field (lint ERR-029). A permanent accuracy corpus (`scripts/score_mappings.py`) pins the mapping field-for-field against author-curated ground truth. See [references/cloud-mapping.md](references/cloud-mapping.md).
- Process / command execution (new in 1.6.1) -- the recommended `xdm.*.process.*` set for endpoint / EDR process starts, command executions and script runs (WARN-044), including AAA / network-device command accounting (a TACACS+ `cmd=` record), which is a command execution mapped to `xdm.target.process.command_line` with operation `OPERATION_TYPE_AUDIT` -- not authentication.
- AAA gateways -- TACACS+ / RADIUS / Cisco ISE syslog, with the correct source/target topology, UPN-shaped identities, and command-accounting routed to the process family.
- Install-blocking constructs the platform rejects with no useful error (all three isolated by live-tenant bisection, all measured at zero false positives against a corpus of shipped rules). ERR-031: an `XDM_CONST` member that does not exist -- the closed lists LOOK like well-known external enumerations and are not, so syslog severity's eight levels map onto only five `LOG_LEVEL` members (no EMERGENCY, no ALERT, no DEBUG: floor 0-1 to CRITICAL and 7 to INFORMATIONAL) and there is no logout verb at all. ERR-032: a `regextract` with more than one capturing group, which is also dead weight since `arrayindex()` reads one. ERR-033: a lookahead or lookbehind -- this engine does not support lookaround and does not say so, the query HANGS rather than failing, so the caller sees a timeout that reads like a crash in whatever ran it. ERR-034: an UNQUOTED read of a raw column whose NAME is a query-language construct -- the READ is the fault, not the assignment, and the `tmp_` convention is no defence because it protects the name a rule creates rather than the name it reads. The escape is a backtick (`` `target` ``), which 328 shipped upstream rules use and never omit, and which the check therefore accepts; renaming the column at source is better where you control what writes it. Which names qualify was derived from a corpus rather than from how SQL-ish a word looks, and the guess is a bad guide: `timestamp` and `dst` are read bare in shipped rules and are demonstrably not reserved, while `contains` and `call` turn out not to be column reads at all (1428 of `contains`'s 1429 occurrences are the operator), so they are excluded for want of evidence rather than because they were measured safe -- a distinction an earlier statement of this got wrong. `in` IS reserved as of 1.9.1: it was held out on the belief that flagging it would fire on every `action in (...)`, which measurement refutes, because the read patterns only match in VALUE position and the operator follows an identifier -- 570 operator uses in the corpus, zero matched, against 9 backticked reads. Its pair `out` is NOT reserved and must not be added on symmetry: it is read bare 8 times in shipped rules, which makes it an ordinary column name like `timestamp` and `dst`. Each of these fails the pack install with an opaque 101704 naming no field and no line.
- Cisco IOS / IOS-XE / WLC message families -- the token is `%FACILITY-SUBFACILITY-SEVERITY-MNEMONIC` and the SUBFACILITY is OPTIONAL, so a three-part pattern silently drops every distributed-system message; a card-prefixed line (`%VIP-3-MSG:SLOT5 %LINK-3-UPDOWN:`) wraps a SECOND complete token and a left-anchored pattern reports the wrapper as the event. Extraction-recipes Recipe 15 carries the verified pattern and the continuation markers. The mapped families: administrative login and lockout (`SEC_LOGIN-*`, `LOGIN-3-TOOMANY_AUTHFAILS`), privilege escalation (`SYS-5-PRIV_*`), port-based access where the FACILITY is the method and `AuditSessionID` is the join key (`DOT1X` / `MAB` authentication versus `AUTHMGR` authorization), ACL logging whose mnemonic SUFFIX declares which fields exist (`SEC-6-IPACCESSLOG{P,DP,NP,RP,S,RL}` -- `S` has no destination and `RL` is not a flow at all), zone-firewall sessions carrying the byte pair (`FW-6-SESS_AUDIT_TRAIL`), and the local command corpus most estates miss (`PARSER-5-CFGLOG_LOGGEDCMD`, `SYS-5-CONFIG_I`, `PARSER-5-HIDDEN`). On WLC the mnemonic states the population -- `AAA_AUTH_ADMIN_USER` versus `AAA_AUTH_NETWORK_USER` -- and `SSHPM` is the SSH POLICY manager, not SSH logins. A wireless client session spans FIVE facilities and three coupled state machines -- APF (802.11 and policy), PEM (policy enforcement and the session in/out record) and MM (roaming between controllers), which the controller itself proves by printing all three states for one mobile in `PEM-3-BADWLANID2`. There is no `AuditSessionID` on the controller: the STATION MAC is the only key joining them, and only `PEM-6-GUESTIN` / `PEM-6-GUESTOUT` / `APF-6-GUEST_ASSIGNED_IP` bind an account to that MAC and an address, so they are what make a wireless session attributable to a person. Find a wireless DETECTION by reading templates, never by reading facility names: `CIDS` logs only its own sensor plumbing and has no successful-shun record at all, `WPS` is 52-of-57 signature-FILE parsing, and the enforcement both exist to perform is emitted by `MM-1-CLIENT_SHUNNED` and the `APF-*` rogue and exclusion records instead. The one WPS detection is split across two records with the offending station in the `_CONT` continuation, so modelling only the first yields a detection with a detector and no subject. See [references/authentication-mapping.md](references/authentication-mapping.md), [references/network-mapping.md](references/network-mapping.md) and [references/process-mapping.md](references/process-mapping.md).

Extraction is aided by [references/extraction-recipes.md](references/extraction-recipes.md) -- verified regex starting points for the common syslog / text / CEF / LEEF shapes -- and by a broad field-anchor index that resolves raw vendor field names to their XDM location. The reference files under [references/](references/) document the parser conformance rules, XDM schema, extraction and transformation patterns, the mapping sets, and known pitfalls. The scripts validate rules instead of relying on a mental checklist.

## Scope

In scope: Data Model Rules (`[MODEL: dataset=..._raw]`).

Out of scope:

- Parsing rules (`[INGEST: ...]`). The references mention them only in validation rules ERR-001 / 002 / 003 / 005 and in the MAPPED-header instruction.
- Reading parser-stamped underscore anchor columns. A MODEL rule must derive every value from the raw dataset columns (or `_raw_log`), never by coalescing a parser-only `_` anchor. Cortex validates a MODEL rule statically against the dataset schema, where parser-only `_` anchors do not exist, so reading one is rejected as an unknown field before any `coalesce()` fallback can run (see ERR-027). Re-derive from raw instead.

The field-anchor synonym index is in scope as a static lookup table. The bundle ships a CLI to query it.

## Called as an instrument

This bundle is usually invoked on its own, but it is also called as an INSTRUMENT by `cortex-content-pack-go-again`, which owns the pack-building process and dispatches the rule-authoring work here. Two standing commitments to that caller live in this section because a promise stored only in the bundle that RECEIVES it is a promise the next session here cannot keep.

The rule-versus-files line. A pack install rejected with an opaque 101704 names no field and no line, so the cause has to be bisected. Causes reachable from the RULE TEXT are this bundle's, and ERR-029 through ERR-034 all arrived exactly that way: a 101704 that bisects to a construct inside the rule belongs HERE and becomes a lint code, which is a standing offer and not a case-by-case favour. Causes that need the pack's FILES are not: the step "if the minimal block also fails, stop bisecting content and bisect FILES" is a statement about pack STRUCTURE, and the case that produced it was a missing `_schema.json` -- a file this bundle never sees and has no vocabulary for. That half stays with the caller, and the ordered bisect procedure with it.

The ERR-034 reserved set is MIRRORED, so a change to it must be announced. `_ERR034_RESERVED` in `scripts/lint_rule.py` is copied by hand into `RESERVED_COLUMNS` in the content-pack bundle's `scripts/preflight_release.py`, which gates the same fault at upload time. Deciding a ninth name is reserved, or retiring one, is therefore not a local edit: message that bundle in the same change, because nothing in either repository can see the other and each side has already believed itself current while holding a list one word behind. The set is pinned at its eight members by `tests/test_lint_rule.py`, whose failure message says who to tell.

EVERY RELEASE MUST DECLARE ITS FIELD IMPACT, because a consumer measures against it. `assets/field_impact.json` answers "did version X change the MEANING of field F" per version, and phase 900 of the content-pack bundle reads it as a precondition: a migration that measures a SECOND field against the still-installed old model is only valid if this bundle did not move that field underneath it. Adding a version to `CHANGELOG.md` without an entry here FAILS the suite, deliberately, because absence in the registry reads as "no impact" and a missing entry is therefore a false negative rather than a gap. Classify every `xdm.` path the entry names into exactly one of `meaning_changed`, `mandatory_changed`, `banned` or `mentioned_only` -- and `mentioned_only` needs a NOTE saying why there is no impact, since the whole point is that a version can discuss a field at length without moving it, which 2.0.2 and 2.0.3 both do for `xdm.event.type`. Query it with `scripts/field_impact.py`; do not hand-read the JSON for a range, because entries are per-version deltas and the union is the answer.

Replying to that caller. It asks for FOUR verdicts per item -- COVERED (with a `file:line` it can open, never a bare "we do that"), TAKING IT (with where it will live), DECLINED (with the reason, which it records verbatim and never re-sends), and CORRECTED (the right rule, when the premise of the question is wrong). Send ONE reply through the channel it asked on rather than a trickle across several, state the version of this bundle you are replying at, and say so in the same reply if you edited a file of theirs. Silence is not a decline there: an unanswered item stays open indefinitely.

## Inputs accepted

1. Required: at least one raw log sample (JSONL, plain JSON, syslog text, CEF -- anything Cortex XSIAM ingests as `_raw_log`). If no sample is supplied, ask for one. Do NOT guess vendor field names.
2. Optional: vendor / product / dataset name. These cannot be derived from the log body; infer them from the product or API title, then flag them as tenant-adjustable in the MAPPED header, naming the three touch-points the reviewer edits: `xdm.observer.vendor`, `xdm.observer.product`, and the `[MODEL: dataset=...]` header.
3. Optional but recommended: a source reference document that describes the fields -- an OpenAPI / JSON Schema spec or API field docs for a JSON / JSONL source, a message / mnemonic reference for a syslog source, a CEF / LEEF extension dictionary, or a column dictionary for CSV / TSV. A sample alone shows the shape but not the meaning: the reference resolves cryptic field names, authoritative datatypes (array vs scalar -- the class of the `xdm.alert.risks` "expected array" defect), enum values (for `XDM_CONST` mapping) and which fields carry the identity / auth story. The skill ASKS for the format-matched reference after profiling (workflow step 3) and proceeds without it when unavailable, recording the basis in provenance (`GOCORTEX_SKILLS_SOURCE_BASIS`). Accept a link (fetch and mine it), pasted text, or a file path.

## Outputs produced

One XQL file (or one code block if inline). The top comment block MUST follow this fixed order for predictability -- SPDX licence always first, the skill-issues pointer always last:

1. SPDX licence header (`SPDX-FileCopyrightText` + `SPDX-License-Identifier`) -- ALWAYS the first lines.
2. Provenance block (`GOCORTEX_SKILLS_*`).
3. Identity: vendor / product / dataset / one-paragraph description.
4. ALERT / EVENT FIELD MAPPING (`->` arrows) + any advisory NOTES + NOT MAPPED list with reasons.
5. REVIEW UNMODELLED query.
6. RAISE SKILL ISSUES pointer.

Then the `[MODEL: ...]` body. The sections in detail:

- A MAPPED-header comment block (mandatory). Vendor / product / dataset / one-paragraph description / Alert-or-Event Field Mapping with `->` arrows / NOT MAPPED list with reasons. The SPDX licence sits at the very TOP of this block (never at the bottom). See [assets/modeling_header_template.xql](assets/modeling_header_template.xql).
- A provenance block (mandatory), emitted verbatim as comment lines directly under the SPDX header so a script can grep it. Fill `GOCORTEX_SKILLS_MODEL` with the model id that authored the rule, `GOCORTEX_SKILLS_SKILL_NAME` / `GOCORTEX_SKILLS_SKILL_VERSION` from this skill's frontmatter (`cortex-platform-xdm-author` / `2.1.0`), `GOCORTEX_SKILLS_SKILL_WARNING_COUNT` with the advisory count from the final `scripts/lint_rule.py` run, and `GOCORTEX_SKILLS_SOURCE_BASIS` with `"spec-backed"` when a source reference (OpenAPI spec, vendor mnemonic doc, ...) informed the mapping or `"sample-only"` when only the raw sample was available (see workflow step 3):
  ```
  // Generated via
  // GOCORTEX_SKILLS_MODEL="<model id>"
  // GOCORTEX_SKILLS_SKILL_NAME="cortex-platform-xdm-author"
  // GOCORTEX_SKILLS_SKILL_VERSION="2.1.2"
  // GOCORTEX_SKILLS_SKILL_WARNING_COUNT="<lint warning count>"
  // GOCORTEX_SKILLS_SOURCE_BASIS="<spec-backed | sample-only>"
  ```
  `scripts/scaffold_rule.py` emits this automatically (name / version from SKILL.md, model and count from the build environment or its self-lint); when hand-authoring, add it yourself.
- The commented REVIEW UNMODELLED query (mandatory), placed as the second-to-last section, so unclassified records are discoverable -- see [references/record-classification.md](references/record-classification.md).
- A RAISE SKILL ISSUES pointer (mandatory), the LAST comment section, inviting the user to report a mis-mapping and include the REVIEW UNMODELLED output:
  ```
  // RAISE SKILL ISSUES -- if this rule mis-modelled something, please open
  // an issue and include the REVIEW UNMODELLED output above:
  //   https://github.com/gocortexio/skills/issues
  ```
- A `[MODEL: dataset=<vendor>_<product>_raw]` block in the three-stage shape: `filter` -> `alter` (extract) -> `alter` (assign).

## Authoring workflow

Full long-form in [references/workflow.md](references/workflow.md). Quick reference:

1. Confirm sample present. If not, ask for one. If the sample is being pulled from a live dataset rather than handed over as a file, it must be STRATIFIED, because a `limit N` pull is not a sample -- it returns whichever records the engine reaches first, which is routinely a tiny unrepresentative minority, and a rule authored from it can look perfect on row count while modelling the wrong shape entirely. Identify the discriminator, count by it (`comp count() by <discriminator>`), then take a few records per group so every shape is represented. If a previous version of this rule exists, also list the `xdm.*` fields it assigns before you start -- see step 12.
2. Profile the sample with `python3 scripts/profile_log.py "<path/to/sample>"`. The script detects the format, walks each record into leaf paths (including `transactions[].http.method` array shapes and `{name, value}` header-pair arrays), infers field types, computes per-field null rates, flags every object-array's discriminator key, attaches ranked XDM candidates per field, and emits a `recommended_pattern` block. JSON worksheet by default; `--format text` for a table. If the script cannot run, fall back to the manual analysis in [references/workflow.md](references/workflow.md) Step 2.
3. Ask for the source reference (soft gate, skippable). After profiling you know `detected_format`, so ask the user once for the format-matched reference that gives the fields meaning, then proceed either way:
   - JSON / JSONL -> the OpenAPI / JSON Schema spec or API field docs (best signal: names, datatypes incl. array vs scalar, enums, which fields are identity / auth).
   - syslog / positional -> the vendor message / mnemonic reference (decodes `%FAC-SEV-MNEMONIC`, positional fields, severity) -- and confirm whether the source can arrive relay-prepended.
   - CEF / LEEF -> the vendor's extension dictionary. CSV / TSV -> the column dictionary.
   Given a link, fetch and mine it; given pasted text or a file, read it. If none is available, proceed with the sample alone -- do NOT block. Record the basis in the provenance block: `GOCORTEX_SKILLS_SOURCE_BASIS = "spec-backed"` when a reference informed the mapping, `"sample-only"` otherwise (a sample-only rule is lower-confidence and worth closer review). If you cannot ask (non-interactive), proceed sample-only and mark it.
4. Pick the extraction pattern. The profiler's `recommended_pattern` already maps `detected_format` and the object-array discriminators onto the decision tree below; confirm it against [references/extraction-patterns.md](references/extraction-patterns.md):
   - `_raw_log` contains JSON string -> A or C
   - `_raw_log` contains syslog / text -> B (if it opens with a `<NNN>` priority token, parse the Stage 0 envelope first -- see [references/syslog-envelope.md](references/syslog-envelope.md) -- then apply B to the payload)
   - `_raw_log` is null, fields are pre-parsed top-level columns -> D
   - object-array with a discriminator -> D' projection per discriminator value
5. Look up XDM targets via `python3 scripts/lookup_anchor.py <vendor_field_name>`. Treat frequency `>= 10` as a strong match, `3 - 9` as the default inclusion gate, `1 - 2` as a candidate-only signal. A `0` result means no anchor PRECEDENT -- the index records what past rules mapped, it is NOT the schema. Do NOT conclude the field has no XDM home from a `0`: first grep [references/xdm-schema.md](references/xdm-schema.md) for the concept (`auth`, `mfa`, `host`, `process`, ...). Only document a field in NOT MAPPED when xdm-schema.md genuinely has no field for it. Example: `mfa_method` returns `0` anchors but `xdm.auth.mfa.method` exists. If the script is unreachable, grep `assets/field_anchors.json` directly.
6. Cross-reference types against [references/xdm-schema.md](references/xdm-schema.md) and apply the transformation patterns in [references/transformation-patterns.md](references/transformation-patterns.md): numeric coercion, companion pairs, array wrapping, banded scoring, one-sided actor mirroring, authentication / MFA mapping (`xdm.auth.mfa.method`, `xdm.auth.is_mfa_needed`, plus the `xdm.event.operation` classification via `XDM_CONST.OPERATION_TYPE_*`). When the profiler flags an authentication event, apply the full mandatory XDM field set in [references/authentication-mapping.md](references/authentication-mapping.md) (the linter raises the advisory WARN-042 for each mandatory field left unmapped). When it flags a network event, apply the mandatory set in [references/network-mapping.md](references/network-mapping.md) likewise (advisory WARN-043). The two are independent, and network is the foundational layer: IDS, WAF, proxy and similar profiles describe a network flow on top of their primary role, and an authentication event carrying the full transport tuple is also a network connection. A dual event takes BOTH sets, with the union of the story tags in ONE `xdm.event.tags = arraycreate(...)`. When it flags a process / command-execution event, apply the recommended `xdm.*.process.*` mapping in [references/process-mapping.md](references/process-mapping.md) (advisory WARN-044). This is a recommended set, not a mandatory story. AAA / network-device command accounting (a `cmd=` record) is a command execution, not authentication: map its command to `xdm.target.process.command_line` with `xdm.event.type` a process value, operation `OPERATION_TYPE_AUDIT` and no outcome, and do NOT tag it `EVENT_TAG_AUTHENTICATION` (only the AUTHEN login and AUTHOR authorization shapes are authentication). Classification is PER RECORD throughout: decide `xdm.event.type` and `xdm.event.tags` on each record via one `if()` over the closed six-member `EVENT_TAG` enum (ending with no default, so an unrecognised record gets blank tags), filter only `_raw_log != null`, and give any record you cannot classify the catch-all (`xdm.event.original_event_type = "GOCORTEX_UNMODELLED"`) plus the commented REVIEW UNMODELLED query, so a `datamodel` search returns the same row count as the raw dataset (see [references/record-classification.md](references/record-classification.md); WARN-045 flags an invented tag, WARN-046 a record-dropping filter).
7. Draft the rule. Three stages minimum: `filter` (null guard), `alter` (extract `tmp_temp` variables), `alter` (assign XDM fields). End with semicolon, no trailing comma. Prefix with the MAPPED-header block. By default, also emit a deterministic `xdm.event.description` summary built with `concat()` over the identifying fields (vendor action, subject, outcome, and so on) -- see [references/transformation-patterns.md](references/transformation-patterns.md) structured event description. This is an ADDITION to the structured XDM fields, never a substitute: map each value to its own field first, then summarise. Do not bury data in the description that belongs in a queryable field.
8. Lint. `python3 scripts/lint_rule.py <rule.xql>` runs the structural, schema-aware, and dataflow checks. It reads the XDM schema and XDM_CONST lists from the references and runs a reach + array-typing pass, so the schema and dataflow checks happen offline. The current code list is not restated here, because a copy of it goes stale silently and this one did: run `python3 scripts/lint_rule.py --list-codes` for the authoritative set with descriptions.
9. Fix earliest-first. The earliest violation is almost always the root cause; the rest are cascade noise. See INFO-012 in [references/parser-idioms.md](references/parser-idioms.md).
10. Re-lint until clean.
11. For a SYSLOG source, prove both arrival forms before emitting. Static lint cannot establish prepend-robustness, so run `python3 scripts/verify_rule.py <rule.xql> <sample> --prepend-check`. It evaluates every record twice -- as supplied and with a relay header prepended -- and exits 1 naming any field whose value differs; the findings sit under the `prepend_check` key of the JSON result, so a caller looking for `differences` or `mismatches` gets an empty list that reads as a false pass. A difference means an extraction is anchored on position rather than on the payload's own token; fix the anchor, do not special-case the second form. This step is mandatory, not optional: a rule that only models the arrival form the sample happened to show is incomplete even when it lints clean.
12. If this rule REPLACES an existing one, diff the emitted field set. Enumerate the `xdm.*` fields the previous version assigned and set-subtract the new version's; anything dropped is a potential break in downstream content, because correlations and dashboards bind to FIELDS, not to datasets. A rewrite that stops emitting a field a correlation reads makes that correlation structurally incapable of firing -- not short of data, unable to match -- and nothing detects it: the rule lints clean, the dataset name is unchanged and row parity is perfect. Mechanical enough to run by hand:
    ```sh
    grep -oE "xdm\.[a-z0-9_.]+" old.xql | sort -u > /tmp/old.f
    grep -oE "xdm\.[a-z0-9_.]+" new.xql | sort -u > /tmp/new.f
    comm -23 /tmp/old.f /tmp/new.f   # fields the new rule no longer emits
    ```
    Either keep the dropped field or update the content that reads it; never drop one silently.
13. Emit the final output.

## Note on intermediate variables

Scratch temporaries use the `tmp_` prefix (`tmp_user`, `tmp_src_ip`, ...). The `_` prefix is reserved by the platform for internal / system-generated fields (`_raw_log`, `_time`, `_message`, ...), so a rule must never CREATE a `_`-prefixed field -- `lint_rule.py` raises ERR-028 if it does (reading `_raw_log` is fine). No explicit `| fields -...` cleanup stage is needed: a MODEL rule surfaces only `xdm.*` fields, so `tmp_` temporaries never reach the datamodel regardless of name; the linter therefore does NOT flag a missing cleanup stage (INFO-006). See [references/extraction-patterns.md](references/extraction-patterns.md) "A note on intermediate variables".

## Output discipline

XQL is a formal language, not a creative-writing surface. Emission discipline matters as much as content correctness:

- Emit the rule, not narration about emitting the rule. No "I'll now draft the rule that...", no "Here is the rule:". The MAPPED header is the documentation; the rule body is the answer.
- Determinism over variety. Two runs against the same log sample should produce the same rule. Pick the highest-frequency XDM target from the field-anchor index (workflow step 4) rather than rotating through alternatives.
- One draft, then lint, then fix-earliest-first. Do not draft three variants and pick one. Draft once, run `scripts/lint_rule.py`, fix the earliest violation, re-lint. See workflow steps 6-9.
- No re-verification loops. Trust the linter. If it reports nothing, the rule is done. If it reports something, fix that one thing.
- No "Not available" enumeration. A vendor field with no XDM home (confirmed against [references/xdm-schema.md](references/xdm-schema.md), not just a zero anchor result) goes in the NOT MAPPED block with a one-line reason. It does NOT generate a list of XDM fields considered and rejected.
- Stop when done. No "let me also check...", no postscript on future improvements. The rule plus its MAPPED header is the deliverable.

These rules exist because the Cortex parser is the ground truth. A rule that reads beautifully but the parser rejects is worse than a rule that reads tersely but the parser accepts.

## Hard rules (do not violate)

- Claim a story only where its mandatory set can be populated. A story tag is a promise that the story's fields mean something on that record, and downstream content queries them as though populated, so tagging a record into a story it cannot fill pollutes the story rather than enriching the record. Before tagging AUTHENTICATION, confirm the record can supply an actor (`xdm.source.user.username` / `upn`); before tagging NETWORK, confirm it can supply a peer (`xdm.source.ipv4`). Two records from one subsystem can differ -- an interface transition has no peer and is a device status change, not a flow. When a record cannot fill a story, give it an honest `xdm.event.type` (`system`, `status`, the subsystem name) plus a descriptive `xdm.event.description`. This is the class of error the linter cannot see: every mandatory field assigned, a clean lint, and the rule still wrong about what the record IS. Never gate `xdm.event.type`, `xdm.event.tags` or any mandatory story field on a facility or subsystem flag -- a facility names the subsystem that spoke, only the mnemonic says what happened -- and never end an `xdm.event.operation` chain on a broad subsystem flag; if nothing specific matches, leave the verb unset. See [references/record-classification.md](references/record-classification.md).
- Terminate a capture on CONTENT, not on the end of the line, wherever the value will be COMPARED rather than displayed. A capture that runs to end-of-line inherits whatever the device put there, and devices commonly emit a trailing space after a CLI command, so `(.+)` keeps it. The field is then populated, non-empty, not the sentinel, and reads correctly in every sample and comp -- and differs from the same value written without the space under exact comparison alone, so a filter for the command returns zero rows rather than an error. Use `(.*\S)`, which is greedy up to a required final non-space: it takes the whole value and drops the trailing whitespace with no trim function. This matters most for `xdm.target.process.command_line` and `xdm.event.original_event_type`, the two natural correlation keys; a free-text description legitimately wants the tail. A capture closed by a delimiter (`command '(.+)'`) is safe against whitespace OUTSIDE the delimiter, but not against whitespace the vendor writes INSIDE it: `management IP Address [198.51.100.28 ]` defeats `\[([^\]]+)\]`, and the linter cannot see it. Where the value will be compared, bound the capture on the CONTENT it must contain (`\[\s*(\d{1,3}(?:\.\d{1,3}){3})\s*\]`) rather than trusting the delimiter. Lint WARN-054 covers the end-of-line form only. See [references/process-mapping.md](references/process-mapping.md).
- Never let letter case qualify a capture; structure must. XQL matches case-insensitively everywhere, so `[A-Z]` also matches lowercase and an uppercase character class does NOT restrict what a group captures. A pattern that lifts an identifier out of free text and relies on the class to mean "this is a vendor tag, not prose" takes whatever token occupies that position, and the field ends up holding a plausible but WRONG value -- a fourth failure mode beyond null, empty and the catch-all sentinel, and the only one no count reveals. Anchor every capture on structure that identifies the token (the `%` sigil and severity digit of `%FACILITY-SEVERITY-MNEMONIC`, a key name, a quote). Where the position genuinely is free text, enumerate the documented vendor tags in an alternation and let anything else fall back; documented vendor tags are not the sample-derived literals WARN-049 forbids. `(?i)` is redundant, case-sensitivity cannot be requested, and `uppercase(x) = x` is always true so no query can test for case either -- auditing a field for case means reading its distinct values outside the query. Lint WARN-052 flags a case-qualified capture with no literal anchor. See [references/xql-language.md](references/xql-language.md).
- Never invent XDM field paths. Every `xdm.*` path must appear in [references/xdm-schema.md](references/xdm-schema.md). If a vendor field has no XDM home, document it in the NOT MAPPED block -- but only after confirming the schema genuinely lacks a field for the concept. A zero anchor-index result is not that confirmation (the index is a precedent table, not the schema).
- Never assign a banned XDM field. A banned field is a REAL Cortex path that belongs to an internal or non-event data model (for example `xdm.*.cloud.source_type`, an XCloud asset attribute), so a MODEL rule that assigns it fails tenant validation with "not part of the selected data model" even though the path looks legitimate. The registry is [assets/banned_fields.json](assets/banned_fields.json) (each entry carries the reason and the correct alternative); the linter blocks any assignment as ERR-029. See [references/banned-fields.md](references/banned-fields.md).
- Never invent `XDM_CONST` values. Closed lists in [references/xdm-const.md](references/xdm-const.md). If no constant matches, OMIT the field and fall back to the String alternative per [references/pitfall-traps.md](references/pitfall-traps.md).
- Always map a const-typed enum over its COMPLETE set, never just the values the sample showed. `xdm.network.http.response_code` is the canonical case: it is const-typed over all 60 HTTP status codes, so cast the status to an integer and render the full crosswalk chain with `python3 scripts/http_status_map.py --render` -- a production source returns codes the sample never contained, and a partial hand-written chain silently drops them (WARN-048). The same principle holds for any const field whose full membership is known.
- For an endpoint / process / file / registry / image event (Linux commands, Windows events, Sysmon, EDR), classify on three fields, not one label: put the raw channel / source label in `xdm.event.type`, the per-record semantic name in `xdm.event.original_event_type`, and the precise verb in `xdm.event.operation` (a member of the 56-strong `XDM_CONST.OPERATION_TYPE_*` enum -- `PROCESS_CREATE`, `IMAGE_LOAD`, `REGISTRY_SET_VALUE`, `FILE_REMOVE`, `EXECUTION`, ...). Do NOT leave `xdm.event.operation` blank when a verb fits, and do NOT stuff a semantic string into `xdm.event.type`. Endpoint events carry blank `xdm.event.tags` -- there is no process story tag -- yet are still fully modelled, so they do NOT take the `GOCORTEX_UNMODELLED` catch-all. See [references/process-mapping.md](references/process-mapping.md).
- For a cloud audit source (AWS CloudTrail, Azure, GCP), DERIVE `xdm.event.operation` from the provider's action-naming convention (CloudTrail `eventName` prefix, Azure `operationName` suffix, GCP `methodName` verb) rather than hardcoding the handful of actions the sample showed -- and keep the raw action in `xdm.event.original_event_type`. Map `xdm.*.cloud.provider` reliably, and set `xdm.*.cloud.service` (`CLOUD_SERVICE_TYPE`) only on a confident known match; when the raw service name maps to no constant, keep it in `xdm.event.type` / `xdm.event.description` or the NOT MAPPED block. NEVER assign `xdm.*.cloud.source_type` -- it is a banned internal-only field (see the banned-field rule below). See [references/cloud-mapping.md](references/cloud-mapping.md).
- Never hardcode a value that came from the sample. A tenant URL path, hostname, IP, ID or product-specific token must NOT be baked into a `contains "..."` branch or an `= "..."` assignment -- that leaks customer-internal data into the rule and does not scale beyond what the sample happened to show (WARN-049). Extract the value dynamically (e.g. `arrayindex(regextract(<field>, "/([^/]+)/"), N)` for a path segment) or keep the raw value in a free-String XDM field. Only hardcode `XDM_CONST` members, the observer vendor / product identity, and well-known vendor-agnostic tokens (`kerberos`, `$`, ...). Never invent a classification the source does not define -- see [references/transformation-patterns.md](references/transformation-patterns.md) "Never hardcode sample-derived values".
- Never use infix arithmetic inside `alter`. Use `add()`, `subtract()`, `multiply()`, `divide()` -- see [references/parser-idioms.md](references/parser-idioms.md) (ERR-012).
- Never assign a raw numeric score to `xdm.alert.severity`. Apply banded scoring per [references/transformation-patterns.md](references/transformation-patterns.md).
- Never echo a log-level word into `xdm.alert.severity`. Words like `warning`, `error`, `notice`, `debug` are syslog levels, not severity bands. Band them to Informational / Low / Medium / High / Critical and map the raw level to `xdm.event.log_level` -- see [references/transformation-patterns.md](references/transformation-patterns.md) log-level vocabulary (WARN-037).
- Always end a categorical if-chain into a free-String XDM field that carries vendor text (`xdm.alert.subcategory`, `xdm.observer.action`, `xdm.alert.original_threat_name`, and similar) with a `tmp_field != null, tmp_field` passthrough, so an unmapped vendor value is preserved rather than silently nulled. This does NOT apply to closed-list `XDM_CONST` targets (omit the default -> null) nor to band-vocabulary fields like `xdm.alert.severity` (floor to a band, never echo raw).
- Always emit the MAPPED header. It is the required prefix for every rule -- see [references/modeling-rules.md](references/modeling-rules.md) and [assets/modeling_header_template.xql](assets/modeling_header_template.xql).
- Always use British English in comments (normalise, analyse, behaviour, colour).

## Mapping decision checklist

Run this before emitting, so the same log maps the same way every time. Each item is a deterministic rule, not a judgement call:

- Outcome only on a real result. Set `xdm.event.outcome` only when the log reports success / failure / blocked. A detection disposition (`alert`, `monitor`, `isolate`) is NOT an outcome -- keep it in `xdm.observer.action` (see transformation-patterns.md "Event outcome").
- Host + IP -> emit the address companion ONLY where both fields describe the SAME entity. When `xdm.<side>.host.hostname` and `xdm.<side>.ipv4` are both set AND name the same host, also set `xdm.<side>.host.ipv4_addresses = if(ip != null, arraycreate(ip), null)` (WARN-038, advisory). They often do NOT name the same host: on a flow-bearing record the hostname is routinely the device that EMITTED the log while the address is a flow endpoint, and hanging the flow address off the emitter yields a populated, non-empty, non-sentinel and WRONG value that every host-based join would silently use. Where the address is the prescribed `""` pad, `arraycreate("")` is junk. Name the entity each field describes before satisfying any companion-field advisory, and where the premise is false, record that in the rule header and leave the array unset -- a visible gap beats an invisible wrong value. See [references/pitfall-traps.md](references/pitfall-traps.md).
- Syslog source -> parse the envelope (Stage 0) before the payload, and make it prepend-robust (HARD RULE). The same source arrives direct and behind a relay that prepends its own `<PRI>` header, so capture the envelope relay-aware (greedy `^.*` -> origin host + origin PRI) and anchor every payload field on its own token, never on `^` -- extraction must be identical for both arrival forms even if the sample showed only one, in ONE rule -- never two rules and never only the observed form. Prove it with `scripts/verify_rule.py --prepend-check`, which fails on any field that changes when a relay header is prepended; this is a required step for every syslog rule, because lint alone cannot establish it. Decode the priority into `xdm.event.log_level` / `xdm.alert.severity` as a fallback under the payload severity -- see [references/syslog-envelope.md](references/syslog-envelope.md) (WARN-040 vendor-anchored header, WARN-041 priority captured but never decoded, ERR-030 prepend-fragile body capture).
- Named asset is a host, cloud object is a resource -- EXCEPT on an authentication event. An OT / ICS asset (`asset=PLC-17`) or server name goes to `xdm.target.host.hostname`; a cloud resource goes to `xdm.target.resource.name` (see pitfall-traps.md). On an authentication event `xdm.target.resource.name` is MANDATORY regardless of the target's kind: it names the device / application / service the principal authenticated TO, set IN ADDITION to the type-correct field and NEVER padded, because a padded target is how an inverted authentication rule (source and target the wrong way round) passes the linter with every mandatory field assigned. WARN-055 flags a placeholder. This is a house convention, recorded with its reason in [references/house-conventions.md](references/house-conventions.md).
- Numeric severity scale -> band both fields. Read the vendor band table, normalise labels to Critical / High / Medium / Low (vendor `Moderate` -> `Medium`), and emit both `xdm.alert.severity` and `xdm.event.log_level` (transformation-patterns.md "Banded numeric scoring").
- Risk / deviation metric -> `xdm.alert.risks`. A ratio or deviation with no typed numeric home is parked in `xdm.alert.risks`, not dropped. The field is an ARRAY of String, so wrap the value with `arraycreate()` -- a bare `concat()` is the scalar-into-array shape WARN-035 catches. If you do drop it, write "intentionally omitted", never "no XDM home".
- Vendor / product / dataset are tenant-adjustable. Infer them, and flag the three touch-points (`xdm.observer.vendor`, `xdm.observer.product`, `[MODEL: dataset=...]`) in the MAPPED header.
- Never bury a value that has a structured home. The description summarises with `concat()` over the fields that matter; it never substitutes for a queryable field, and it NEVER receives the whole payload (`_raw_log` or `to_json_string(...)` -- WARN-039). See also WARN-038 / INFO-013 and failure-modes.md.

## References (load on demand)

- [references/workflow.md](references/workflow.md) -- full step-by-step
- [references/modeling-rules.md](references/modeling-rules.md) -- `[MODEL: ...]` structure
- [references/xql-language.md](references/xql-language.md) -- rule structure, functions, operators
- [references/parser-idioms.md](references/parser-idioms.md) -- ERR-012 through ERR-019, INFO-012
- [references/xdm-schema.md](references/xdm-schema.md) -- 628-field XDM path list
- [references/xdm-const.md](references/xdm-const.md) -- closed-list constants
- [references/banned-fields.md](references/banned-fields.md) -- real Cortex paths a MODEL rule must never assign (internal / non-event data models; enforced by ERR-029 from [assets/banned_fields.json](assets/banned_fields.json))
- [references/extraction-patterns.md](references/extraction-patterns.md) -- A / B / C / D extraction patterns
- [references/extraction-recipes.md](references/extraction-recipes.md) -- verified regex recipes for common syslog / text / CEF / LEEF shapes (advisory starting points for Pattern B; each proven lint-clean and by the verifier)
- [references/syslog-envelope.md](references/syslog-envelope.md) -- Stage 0 transport layer for syslog sources (PRI-anchored host + priority decode)
- [references/transformation-patterns.md](references/transformation-patterns.md) -- coercion, companion pairs, banded scoring, mirroring
- [references/authentication-mapping.md](references/authentication-mapping.md) -- mandatory XDM field set for authentication events (auto-detected; advisory WARN-042, plus WARN-055 for a padded `xdm.target.resource.name`)
- [references/house-conventions.md](references/house-conventions.md) -- the register of places this bundle requires MORE than the XDM schema does, each with the platform position, our reason, and what evidence would retire it
- [references/network-mapping.md](references/network-mapping.md) -- mandatory XDM field set for network events (auto-detected; advisory WARN-043; dual-story tag union)
- [references/process-mapping.md](references/process-mapping.md) -- recommended XDM mapping for process / command-execution events (endpoint / EDR process starts, script runs, AAA command accounting; auto-detected, advisory WARN-044)
- [references/record-classification.md](references/record-classification.md) -- per-record classification of `xdm.event.type` / `xdm.event.tags` over the closed six-member EVENT_TAG enum, and the catch-all that keeps the datamodel row count equal to the raw count (advisory WARN-045 / WARN-046)
- [references/cloud-mapping.md](references/cloud-mapping.md) -- cloud audit-log mapping (AWS CloudTrail, Azure Activity / Entra sign-in, GCP Cloud Audit): provider / region / account entity, the per-provider action -> `OPERATION_TYPE` verb conventions, outcome derivation, cloud / saas / auth classification, and nested-JSON extraction
- [references/mitre-mapping.md](references/mitre-mapping.md) -- MITRE ATT&CK into the `xdm.alert.mitre_techniques` / `mitre_tactics` arrays: direct id/name mapping via the full authoritative crosswalk (`assets/mitre_crosswalk.json`, resolved by `scripts/mitre_map.py`), and high-confidence keyword -> tactic fuzzy mapping that collects EVERY matched tactic (multi-match); auto-detected by the profiler
- [references/pitfall-traps.md](references/pitfall-traps.md) -- non-existent paths, confused pairs
- [references/compatibility-notes.md](references/compatibility-notes.md) -- `_gc_raw` caveats, deprecated fields
- [references/failure-modes.md](references/failure-modes.md) -- "if you see this in your draft, stop and do that" empirical notes
- [references/worked-examples.md](references/worked-examples.md) -- index of sixteen end-to-end log-to-rule walkthroughs (patterns A-D', the syslog envelope, the authentication / network / AAA stories, endpoint telemetry for Sysmon / Windows 4688 / Linux, Windows logon + Kerberos, cloud audit for AWS CloudTrail / Azure / GCP, and a management-plane application log whose correct handling is mostly about what NOT to claim); each lives in its own file under [references/worked-examples/](references/worked-examples/) so you load only the pattern in front of you

## Scripts

All scripts are Python 3.9+ stdlib only -- no Node, no `pip install`, no network access. Run from the bundle root. The typical loop is profile -> scaffold -> lint -> verify.

- `python3 scripts/profile_log.py <sample>` -- static profiler for a raw vendor log sample. Detects format (JSON, JSONL, CEF, LEEF, syslog, key=value, CSV, TSV), walks each record into leaf paths (with `[]` markers and per-name entries for `{name, value}` header-pair arrays), infers types, computes per-field null rates, flags object-array discriminators, attaches ranked XDM candidates, and recommends an extraction pattern. JSON worksheet by default; `--format text` for a table. Exits 0 on success, 1 on argument error, 2 on unreadable or unparseable input.
- `python3 scripts/scaffold_rule.py <worksheet.json>` -- turns a profiler worksheet (path or `-` for stdin) into a complete, lint-clean starter MODEL rule: MAPPED header, extraction stage, and an XDM drain wired from the worksheet's anchor candidates. `--vendor` / `--product` / `--dataset` set the identity. Self-gates through the linter before printing.
- `python3 scripts/lookup_anchor.py <vendor_field_name>` -- ranked XDM target candidates from the field-anchor index. `--reverse <xdm.path>` lists the vendor synonyms that fill a target (top-down authoring); `--related <xdm.path>` lists companion / mirror fields. The index ships under `assets/field_anchors.json`.
- `python3 scripts/xdm_const_mapper.py --field <xdm.path> --values a,b,c` -- emits the `if()`-chain mapping vendor values to the field's XDM_CONST members (never invents a constant). `--banded` emits the paired severity / log-level chains for a score column.
- `python3 scripts/mitre_map.py --kind technique --ids T1078,T1059` -- maps MITRE technique / tactic IDs or `--names` to `XDM_CONST.MITRE_*` constants and emits the `arraymap` chain. Validated against the documented MITRE lists; unmapped inputs are reported, not invented.
- A rule file may hold SEVERAL `[MODEL: ...]` blocks, one per dataset, which is the normal shape for a pack. Both `lint_rule.py` and `verify_rule.py` split such a file and analyse each block on its own, because every check reasons about a single rule: a temp unused in its own block must not look used because a later block reuses the name, and a field assigned once per block must not look assigned repeatedly. Lint findings carry the dataset they belong to; the verifier reports a labelled section per block, and `--dataset <name>` selects one.
- `python3 scripts/lint_rule.py <rule.xql>` -- the rule linter. Three families of check: structural and parser-conformance (semicolons, trailing commas, self-reference, infix arithmetic, sibling reference, the reserved `_` temp namespace), schema-aware checks read from the references (invented paths, banned internal-only fields, quoted `XDM_CONST`, array-vs-scalar shape, the story mandatory sets, syslog envelope and prepend-robustness, the closed `EVENT_TAG` enum, record-dropping filters, sample-derived literals), and dataflow checks over the rule's temps (a `tmp_` that never reaches an `xdm.*` assignment is a hard block on EVERY dataset, since Cortex rejects an unused field regardless of suffix). Exits 0 on clean, 1 if any error-severity violation fires; warnings and info never change the exit code. Run `python3 scripts/lint_rule.py --list-codes` for the current code list with descriptions -- the linter is the single source of truth for it, so this file does not restate it. INFO-006 (cleanup stage) is intentionally not emitted -- see the note above.
- `python3 scripts/verify_rule.py <rule.xql> <sample.json>` -- evaluates the rule against the sample offline and prints the resulting `xdm.*` map per record, so you can confirm behaviour without a tenant. `--expect <expected.json>` diffs against expected output. `--prepend-check` evaluates every record twice, as supplied and with a relay header prepended, and exits 1 naming any field whose value differs -- the mandatory proof that a syslog rule models both arrival forms. `--coverage` summarises how many records populated each `xdm.*` field and how many of those values were the empty string, flagging any field that is predominantly or entirely empty -- the signature of a capture that matches every record and captures nothing (see the doubled-quote trap in [references/extraction-recipes.md](references/extraction-recipes.md)). A record carrying an explicit `_raw_log` column has that value bound; only a record without one has `_raw_log` synthesised from its JSON text. Unsupported constructs are reported, not guessed.
- `python3 scripts/field_impact.py --field <xdm.path> --from <version> --to <version>` -- answers whether THIS BUNDLE changed the meaning of a field between two of its own releases, from `assets/field_impact.json`. For a consumer measuring a field against a model authored under a different version: exit 0 means safe to measure across the range, 1 means the meaning moved (or the field was banned) and the values are not comparable, 2 means the registry cannot answer. `--version <v>` prints one release's whole entry; `--json` for machine use. The range is a UNION of per-version deltas and `--from` is EXCLUSIVE, which is the part a grep over one changelog entry gets wrong: `xdm.auth.service` reads clean across 1.9.1 -> 2.0.0 and moved across 1.8.24 -> 2.0.0, because 2.0.0 only restates 1.9.0's reversal.
- `python3 scripts/score_mappings.py --report` -- runs the author-curated mapping-accuracy corpus (`tests/corpus/mapping_matrix.json`: sample event -> the CORRECT XDM output) through the worked-example rules and reports per-source and overall field-for-field accuracy. The ground truth is authored from provider docs + the XDM schema, so the score measures correctness, not agreement with any content pack.

If the scripts cannot run (no Python available), treat the markdown references as the authoritative checklist: walk [references/parser-idioms.md](references/parser-idioms.md) (ERR-012 through ERR-019, plus idioms (xi) / (xii)), [references/modeling-rules.md](references/modeling-rules.md) (validation checklist), and [references/pitfall-traps.md](references/pitfall-traps.md) before emitting the rule.

## Bundle integrity tests

The bundle ships Python stdlib tests under [tests/](tests/). They cover JSON validity, SPDX-header presence, frontmatter shape, doc-to-schema consistency for every `xdm.*` and `XDM_CONST.*` cited in the references, ASCII-only and no-emphasis hygiene, the MAPPED-header template's required rows, and the linter's behaviour on a set of fixtures. Run from the bundle root:

```sh
python3 -m unittest discover -v -s tests
```

Python 3.9+ stdlib only. These tests cover the bundle itself. The path for linting user XQL rules is `python3 scripts/lint_rule.py`. See [tests/README.md](tests/README.md) for what each test guards.