v1.0.0 to v2.0.0
192 added, 32 removed. Audit A to A.
---
name: exchange-multicast-feed-handling
- description: Use when connecting to co-located exchange gateways (CME MDP 3.0, NASDAQ
- MoldUDP64) to handle dual A/B UDP multicast channels, re-sequence out-of-order packets,
- and issue TCP historical gap-fill re-transmission requests.
- domain: algorithmic-trading
- subdomain: real-time-architecture
+ description: Use when building or auditing the sequencing layer of a co-located UDP
+ multicast feed handler (CME MDP 3.0, Nasdaq MoldUDP64, Eurex T7 EMDI) - arbitrating
+ the redundant A/B lines, re-sequencing out-of-order datagrams, holding a gap for an
+ arbitration window before declaring loss, and escalating through the venue's
+ retransmission and snapshot recovery tiers.
+ domain: Venue Integration & Protocols
+ subdomain: Multicast Line Arbitration & Feed Recovery
tags:
- - real-time-architecture
- udp-multicast
- - cme-mdp
+ - line-arbitration
+ - cme-mdp3
- moldudp64
- - co-location
+ - eurex-t7-emdi
- packet-resequencing
- - gap-fill
+ - feed-recovery
+ - co-location
brokers_frameworks:
- - Multicast Feed Handler
- - Python Socket Engine
- version: "1.0.0"
+ - CME MDP 3.0
+ - Nasdaq MoldUDP64
+ - Eurex T7 EMDI
+ - Python dataclasses / enum
+ version: "2.0.0"
author: algo-trading-skills-contributors
license: Apache-2.0
---
## When to Use
- Invoke this skill when operating co-located ultra-low-latency trading servers receiving raw UDP multicast feeds from derivative and equity exchanges (e.g., CME MDP 3.0 Channel A/B, NASDAQ MoldUDP64, Eurex ETI/EMDI). Unlike TCP/WebSockets, UDP multicast does not provide native delivery guarantees or re-transmission. This skill arbitrates between redundant Channel A and B multicast streams, re-sequences out-of-order packets, and requests missing sequence ranges from TCP historical re-transmission servers.
+ Use this skill when writing or reviewing the **sequencing layer** of a feed handler that
+ consumes redundant UDP multicast lines from an exchange. It covers the decisions that a
+ handler gets wrong in ways that either corrupt a book silently or flood a venue with
+ recovery requests: which copy of a duplicated datagram to keep, how long to wait before
+ calling a missing packet lost, how many recovery requests one gap may generate, what to
+ do when recovery comes back incomplete, and how to survive a sequence-number restart.
+ The recovery model here is a three-tier hierarchy, and the ordering is the venue's, not a
+ preference:
+
+ 1. **A/B line arbitration.** CME sends every packet on both Feed A and Feed B and states
+ the two feeds "should be used for arbitration"; Eurex replicates onto Service A and
+ Service B and advises participants to join both. This tier costs nothing and repairs
+ most loss.
+ 2. **Retransmission**, where the venue offers one — and the transports differ. CME uses a
+ TCP replay session; Nasdaq MoldUDP64 uses a **UDP unicast** Request Packet to a
+ Re-request Server; **Eurex T7 has no retransmission service at all**.
+ 3. **Snapshot resynchronization** — CME Market Recovery (which CME recommends, together
+ with Natural Refresh, as the *primary* recovery option), or the Eurex depth-snapshot
+ channel.
+
+ Every protocol fact in this skill is cited to a primary specification in
+ `references/standards.md`. Re-verify against the document version your firm is certified
+ against before relying on it.
+
+ ## When NOT to Use
+
+ - **As a socket layer.** This module owns no sockets and issues no IGMP joins. Binding,
+ `IP_ADD_MEMBERSHIP`, `SO_REUSEADDR`, source-specific multicast filters, NIC ring sizing
+ and CPU pinning are the caller's, and a datagram's arrival line must be determined by
+ the caller (a single socket joined to both groups cannot tell A from B). It is a
+ deliberate split: the sequencing decisions are the part that is subtle and testable.
+ - **As a message decoder.** Payloads are opaque bytes here. Parsing the CME SBE packet
+ header, the MoldUDP64 20-byte header and Message Blocks, or Eurex FAST/FIX belongs to
+ `binary-protocol-parsing-for-low-latency-feeds`.
+ - **For per-product gap tracking.** Gaps are tracked in one packet-level sequence space
+ per channel. Eurex describes an "optimistic" alternative that decodes to message level
+ and recovers only products of interest; that needs a decoder and is out of scope here.
+ - **As a latency-optimised production handler.** This is a correctness reference in
+ Python — dict-based buffer, per-packet allocation, `Enum` comparisons. A colocated
+ handler needs preallocated, cache-friendly structures.
+ - **For in-band recovery feeds.** Eurex T7 MDI and Euronext MDG Lite carry snapshots and
+ incrementals on one feed with different sequencing rules. For Euronext Optiq see
+ `euronext-optiq-market-data-integration`.
+
## Prerequisites
- - Multicast IP groups and port numbers for Channel A and Channel B (e.g., `233.252.8.1:14001`).
- - Exchange TCP historical re-transmission server endpoint.
+ - The **A and B multicast group/port pairs** for the channel, and a socket layer that
+ tags each datagram with the line it arrived on.
+ - The **packet sequence number** decoded from the venue's packet header — CME MsgSeqNum
+ (uInt32, little-endian), MoldUDP64 Sequence Number (8 bytes, big-endian), Eurex
+ PacketSeqNum. Sequence spaces are per channel; one handler instance per channel.
+ - For MoldUDP64, the header's **Message Count** as well, because that feed numbers
+ messages rather than packets.
+ - An **arbitration window** measured for your own network. There is no safe default:
+ Eurex publishes the maximum expected recovery interval as `MDRecoveryTimeInterval`
+ (tag 2565) in the T7 RDI Product snapshot; for other venues, measure the A-versus-B
+ arrival skew on your own cross-connects. The constructor requires it.
+ - A configured path to whichever recovery tiers your venue offers, plus a snapshot
+ consumer — snapshot recovery is the only tier every venue here provides.
## Workflow
- 1. **Bind Dual A/B Multicast Sockets**:
- - Join UDP multicast groups for Channel A and Channel B.
+ 1. **Ingest both lines into one handler instance**:
+ - Call `ingest_packet(channel, sequence_id, payload, message_count)` for every datagram
+ from A and from B. The first copy to arrive is processed; the second is reported
+ `DUPLICATE`.
+ - **Decision point — dedup must cover packets that are still ahead of the stream.**
+ A twin that arrives while its sequence is sitting in the re-sequencing buffer is
+ still a duplicate. Comparing only against the expected sequence misses it, and the
+ packet then gets buffered twice and re-triggers recovery.
- 2. **Deduplicate & Re-Sequence Packets**:
- - Track expected sequence $S_{\text{expected}}$. If packet $S$ arrives on Channel A or Channel B, process immediately.
- - Buffer future out-of-order packets ($S > S_{\text{expected}}$).
+ 2. **Classify a low sequence number before discarding it**:
+ - **Decision point — a large backward jump is a restart, not a duplicate.** CME resets
+ MsgSeqNum weekly and restarts it on a Channel Reset (35=X, 269=J, with 1180-ApplID);
+ MoldUDP64 opens a new Session. The handler reports `RESET_SUSPECTED` and refuses to
+ apply the packet, because from sequence numbers alone a restart and a stale replayed
+ packet are indistinguishable. Confirm from the venue's in-band signal, then call
+ `reset_sequence()`. A handler that files every low sequence under "duplicate" goes
+ deaf for the rest of the week.
- 3. **Issue TCP Historical Re-Transmission Request**:
- - If both Channel A and Channel B drop packet $S$ (sequence gap detected), send TCP gap request for missing range $[S_{\text{expected}}, S-1]$.
+ 3. **Hold an out-of-order gap before declaring loss**:
+ - Buffer the early packet, open one gap at `[expected .. seq-1]`, and start the
+ arbitration window. Return `BUFFERED`, not a recovery request.
+ - **Decision point — every lost packet starts life as a delayed one.** Multicast
+ reorders routinely and the twin is normally microseconds behind on the other line.
+ Requesting recovery on the first reordered datagram converts ordinary reordering
+ into request volume the venue has capped.
+ - While the gap stays open, later out-of-order packets **extend its range without
+ restarting the timer** — Eurex states that recovery already pending must not have
+ its timer reset — and yield no second request.
- 4. **Reconcile Gap & Drain Buffer**:
- - Ingest re-transmitted TCP packets and drain buffered multicast frames in sequential order.
+ 4. **Escalate exactly once when the window elapses**:
+ - `ingest_packet` and `poll_recovery` both age the gap; the crossing call returns one
+ `RecoveryRequest`. **Poll on a timer as well as on ingest**: after a loss the feed
+ can go quiet, and a handler that only evaluates gaps when packets arrive never
+ escalates.
+ - **Decision point — size the request against the venue's limits.** CME caps a replay
+ at 2000 packets, keeps only 24 hours, and processes one Market Data Request per
+ login/logout cycle. MoldUDP64 returns only the messages that fit one datagram, so a
+ large range needs several requests. If the range exceeds what the venue will serve,
+ go to the snapshot tier instead.
+ 5. **Apply recovered data ahead of queued data**:
+ - `apply_recovery_packets()` releases the recovered packets first and then drains the
+ buffered real-time packets behind them, which is the order CME instructs.
+ - **Decision point — a partial fill does not close the gap.** The handler narrows the
+ gap, re-arms the timer, and reports `is_gap_closed=False` with the outstanding range.
+ Treating a partial response as success leaves a permanent hole no later update
+ repairs.
+
+ 6. **Fall back to the snapshot when arbitration cannot repair the stream**:
+ - Overflowing `max_buffered_packets` latches `requires_resynchronization`. So does any
+ gap the venue will not replay. Call `resynchronize(next_sequence)` from the snapshot.
+ - **Decision point — stop quoting while that flag is set.** The missing packet may
+ have deleted the level you are quoting against, and "resume when updates look normal
+ again" resumes against a corrupt book.
+
> Full procedure: see `references/workflows.md`.
> Standards reference: see `references/standards.md`.
> Printable pre-flight checklist: see `assets/checklist.md`.
## Common Pitfalls
- - **Single Multicast Channel Dependency**: Reading only Channel A without joining Channel B, missing instantaneous packet recovery when Channel A drops a packet.
- - **Unbounded Out-of-Order Buffer Growth**: Failing to cap the out-of-order packet queue during sustained network multicast drops.
- - **Ignoring IGMP Group Joins**: Forgetting to issue IGMP `IP_ADD_MEMBERSHIP` socket options, preventing NIC hardware from receiving multicast traffic.
+ - **Requesting recovery on the first out-of-sequence datagram.** UDP delivers out of
+ order as a matter of course. Eurex's guidance is to start a timed operation and cancel
+ it if the missing packet arrives in reasonable time; CME reserves TCP replay for
+ small-scale recovery and says it should only be used if other options are unavailable.
+ An immediate request also races the twin on the other line, which usually arrives first
+ and makes the request pointless.
+ - **Emitting one request per out-of-order packet.** With a gap open at 101 and packets
+ 103, 104, 105 arriving, a handler that re-derives the range each time issues three
+ overlapping requests. One gap gets one request; later packets extend its range.
+ - **Deduplicating only against the expected sequence.** That catches the twin of a packet
+ already processed, but not the twin of one still buffered — which is exactly the case
+ A/B redundancy exists to produce during a gap.
+ - **Treating a weekly or channel reset as a stream of duplicates.** CME resets MsgSeqNum
+ weekly. A handler that silently drops everything below its expected sequence discards
+ 100% of the following session while logging nothing above debug level.
+ - **Assuming one sequence number per packet on MoldUDP64.** The header carries the
+ sequence of the *first* message plus a Message Count, and the rest are implicitly
+ numbered; the next expected sequence is `seq + message_count`. Advancing by one
+ manufactures a gap on every multi-message packet.
+ - **Assuming a TCP historical server exists.** It is venue-specific: CME has TCP replay,
+ MoldUDP64 re-requests over UDP unicast, and Eurex T7 has no retransmission service —
+ a packet lost on both Eurex services is recoverable only from the snapshot channel.
+ - **Closing a gap on a partial recovery response.** MoldUDP64 returns only the messages
+ that fit one datagram and CME caps a replay at 2000 packets, so an incomplete response
+ is the normal case for a large gap, not an error case.
+ - **Letting the re-sequencing buffer grow without a cap.** A sustained outage otherwise
+ turns a feed handler into a memory-exhaustion incident. A full buffer is also the
+ signal that arbitration has failed and only a snapshot will restore the book.
+ - **Timing the arbitration window on the wall clock.** `time.time()` can step backwards
+ under NTP correction, which either fires recovery instantly or suspends it. Use a
+ monotonic clock.
+ - **Sharing one handler across channels or instruments.** Sequence spaces are per channel
+ (CME per channel, Eurex per SenderCompID per multicast address/port); interleaving two
+ of them produces continuous false gaps.
## Verification
- - Simulate dual UDP Channel A/B packets (seq 100 on A, seq 101 on B) and verify seamless merged sequence stream.
- - Simulate packet gap on both channels and verify TCP re-transmission request trigger.
- - Run `python scripts/test_multicast_handler.py` and confirm 100% pass rate.
+ - Instantiate `ExchangeMulticastFeedHandler(100, arbitration_window_s=0.010)` with an
+ injected clock. Feed seq 100 on A, then 100 on B, then 101 on B; verify `PROCESSED`,
+ `DUPLICATE`, `PROCESSED` and `expected_sequence == 102`.
+ - Feed 100 then 102 on A, advance the clock by half the window, then feed 101 on B;
+ verify both 101 and 102 are released in order, the gap closes, and
+ `recovery_requests` is still empty — the twin line did its job.
+ - Feed 100 then 103 on A and verify `BUFFERED`, `missing_range == (101, 102)` and no
+ recovery request. Advance the clock by the full window, then feed 104, 105, 106, 107;
+ verify exactly **one** `RecoveryRequest` starting at 101 exists.
+ - Feed 100 then 103, then repeatedly feed further out-of-order packets inside the window;
+ verify `pending_gap.detected_at` never moves.
+ - With a gap open, call `apply_recovery_packets` with only 101 of `[101, 102]`; verify
+ `is_gap_closed` is False, `outstanding_gap == (102, 102)`, 103 is still buffered, and
+ the gap re-arms so a second request can be issued.
+ - Start at sequence 4,000,000 and feed sequence 1; verify `RESET_SUSPECTED`, nothing
+ processed, and `expected_sequence` unchanged. Then `reset_sequence(1)` and verify the
+ same packet is processed.
+ - MoldUDP64: feed seq 1 with `message_count=3`; verify `expected_sequence == 4`. Feed a
+ straddling recovery packet at seq 2 with `message_count=4`; verify `PARTIAL_OVERLAP`
+ and `first_new_message_index == 2`.
+ - With `max_buffered_packets=3`, buffer three packets and feed a fourth; verify
+ `DROPPED_BUFFER_FULL` and `requires_resynchronization`, then clear it with
+ `resynchronize()`.
+ - Negative checks: constructing without `arbitration_window_s`, a negative
+ `initial_sequence`, a negative `sequence_id`, a boolean sequence number, a non-`bytes`
+ payload, and `message_count=0` must each raise.
+ - Run `python scripts/test_multicast_handler.py` and confirm a 100% pass rate.
## Related Skills
- - `binary-protocol-parsing-for-low-latency-feeds`
- `sequence-number-gap-detection-for-feeds`
+ - `binary-protocol-parsing-for-low-latency-feeds`
+ - `market-data-snapshot-plus-delta-reconciliation`
+ - `nasdaq-totalview-itch-feed-parsing`
+ - `euronext-optiq-market-data-integration`
+ - `cme-globex-futures-api-integration`
+ - `eurex-market-data-and-order-api`
- `clock-synchronization-ptp-for-trading-hosts`
- ---