git:20260927.e328ad1 to git:20260927.6027bc7

- # LemonGateway AGENTS.md

-

- LemonGateway is Lemon's execution runtime. It handles concurrency-slot scheduling,

- per-conversation launch isolation, run lifecycle, streaming bus events, cancellation,

- and session resumption. Gateway-owned transport, SMS, voice, and command startup is

- off by default and starts only when `:gateway_ingress_enabled` is set for

- `:lemon_gateway`. Those ingress surfaces are gateway-owned by design; see the

- "Transports" section of `README.md` and decision D2 in `docs/platform-split.md`.

-

- ## Executor Contract

-

- Gateway has one configured singleton executor: `CodingAgent.Executor`. It is selected

- through `:lemon_gateway, :executor`; `LemonGateway.Executor` validates its readiness

- at startup and before runtime submission. Every gateway run uses that executor and has

- the fixed top-level provenance `engine: "lemon"`. Delegated task records retain their

- own subagent identity; neither provenance field selects Gateway execution.

-

- This is an intentional breaking removal of the old Gateway engine platform:

-

- - Do not add `LemonGateway.Engine`, `EngineRegistry`, `EngineCase`, `Echo`, gateway

- engine shells, or engine-registration code.

- - Do not add `engine`, `default_engine`, `engine_preference`, or

- `[gateway.engines.<id>]` configuration. Gateway configuration cannot select a

- different executor.

- - `EngineInfoBridge` retains only its transport-registry and gateway-config

- capabilities. It no longer exposes Gateway engine enumeration.

- - `LemonCore.ResumeToken` remains the shared resume-token

- mechanism for the native executor and delegated tasks.

-

- ### Migration choices

-

- Choose the extension boundary that matches the work rather than restoring a top-level

- Gateway engine:

-

- | Need | Use |

- | --- | --- |

- | Connect a model/API to an agent | a `LemonAi` provider |

- | Add in-process agent capability | a `CodingAgent` tool |

- | Delegate bounded work to a subagent | the native `task`/`agent` tools (in-process `CodingAgent.Session`) |

-

- Subagents are native in-process `CodingAgent.Session` executions coordinated by

- `CodingAgent.Coordinator`; they never select or replace the Gateway executor.

- There are no vendor CLI task runners.

-

- ## Quick Orientation

-

- **Entry point:** `LemonGateway.Runtime.submit_execution(%LemonCore.ExecutionCommand{})`.

- The removed `LemonGateway.Runtime.submit/1` compatibility path must not return.

-

- **Core loop:**

-

- ```

- Router -> LemonCore.ExecutionCommand -> Runtime -> ExecutionRequest

- -> Scheduler -> ThreadWorker -> Run -> configured Executor -> bus events

- ```

-

- Gateway owns slot allocation, worker/process lifecycle, and safety rails.

- `ThreadWorker` is a dumb per-conversation launcher; `EngineLock` is

- failure-isolation defense in depth, not product queue semantics.

- Scheduler slot requests from `ThreadWorker` carry a generation token, so timeout

- retries cancel the previous generation before requesting another slot. Run-start

- attempts remain bounded across slot grants; permanently unstartable requests emit

- one structured terminal completion and are removed before the worker advances.

- `EngineLock` may transfer ownership on explicit release or confirmed owner death,

- never merely because a live owner is older than the configured observation threshold.

-

- ### Execution flow

-

- 1. Router-owned `SessionCoordinator` decides collect/followup/steer/interrupt

- semantics and submits queue-semantic-free execution commands.

- 2. `Runtime` creates a gateway-private `ExecutionRequest`.

- 3. `Scheduler` routes the request by the router-supplied conversation key and grants

- a concurrency slot.

- 4. `ThreadWorker` starts a `Run` through `RunSupervisor`.

- 5. `Run` acquires `EngineLock`, invokes the configured `LemonGateway.Executor`, and

- broadcasts native-executor events and deltas on `LemonCore.Bus("run:<run_id>")`.

- 6. On completion, `Run` stores resume state, releases the lock and slot, and notifies

- the router-facing observers.

-

- ## Key Files

-

- ### Execution pipeline

-

- | File | Module | Responsibility |

- | --- | --- | --- |

- | `lib/lemon_gateway.ex` | `LemonGateway` | Public execution submission and health facade |

- | `lib/lemon_gateway/runtime.ex` | `Runtime` | Implements `LemonCore.EngineRuntime`; submission, cancellation, and lookup |

- | `lib/lemon_gateway/executor.ex` | `Executor` | Singleton executor boundary and readiness validation |

- | `lib/lemon_gateway/execution_request.ex` | `ExecutionRequest` | Gateway-private request with no queue-mode semantics |

- | `lib/lemon_gateway/scheduler.ex` | `Scheduler` | Slot allocator and thread routing |

- | `lib/lemon_gateway/thread_worker.ex` | `ThreadWorker` | Per-conversation slot waiter and launcher |

- | `lib/lemon_gateway/run.ex` | `Run` | Executor lifecycle, bus events, steering, cancellation, and locks |

- | `lib/lemon_gateway/engine_lock.ex` | `EngineLock` | Per-session mutex with FIFO wait queue, owner-death release, and over-age live-owner telemetry |

- | `lib/lemon_gateway/event.ex` | `Event`, `Event.Delta` | Lifecycle-event constructors and streamed delta struct |

-

- ### Configuration and ingress

-

- Gateway config comes from the canonical TOML `[gateway]` section through

- `LemonCore.GatewayConfig`; transport config is read only from that path. The

- `LemonGateway.Config` application override is test-only. The executor module belongs

- to application configuration, not TOML engine selection.

-

- | File | Module | Responsibility |

- | --- | --- | --- |

- | `lib/lemon_gateway/config.ex` | `Config` | Typed runtime configuration access |

- | `lib/lemon_gateway/config_loader.ex` | `ConfigLoader` | Canonical TOML loading and parsing |

- | `lib/lemon_gateway/binding_resolver.ex` | `BindingResolver` | Resolves cwd and agent metadata for gateway ingress |

- | `lib/lemon_gateway/transport_registry.ex` | `TransportRegistry` | Gateway-ingress transport registration and enablement |

- | `lib/lemon_gateway/transport_supervisor.ex` | `TransportSupervisor` | Starts enabled gateway-native transports |

- | `lib/lemon_gateway/ingress_supervisor.ex` | `IngressSupervisor` | Gateway-owned transport, command, SMS, and voice startup |

-

- Gateway-native transports normalize requests to `LemonCore.RunRequest` and submit

- through `LemonCore.RouterBridge`; they must not invoke router internals directly.

- New user-facing channels belong in `lemon_channels` unless they need a synchronous or

- live gateway ingress surface.

-

- ## Adding Extensions

-

- ### Add a model provider

-

- Implement the appropriate `LemonAi` provider contract when a new model/API needs to

- serve agent turns. Provider selection stays in the agent/model layer; it does not

- change Gateway execution or its singleton executor.

- ### Add a CodingAgent tool

- Add an in-process capability as a `CodingAgent` tool. Gateway-specific tools are

- assembled by the native executor's session runner; they are not a Gateway plugin

- mechanism. Keep tool action details lossless, including safe failure metadata such as

- `error_type`, `timeout_ms`, and `exit_code`.

- ### Add a delegated subagent

- Delegated work runs natively in-process. Add a subagent persona through

- `CodingAgent.Subagents` (`.lemon/subagents.json` or `~/.lemon/agent/subagents.json`)

- and invoke it through the `task`/`agent` tools; the child run is a

- `CodingAgent.Session` coordinated by `CodingAgent.Coordinator`. Do not wrap

- external CLIs as Gateway executors or as task runners — vendor CLI subprocess

- runners (Claude Code, Codex, Kimi, OpenCode, Pi) were removed.

- ### Add a transport

- Prefer `lemon_channels` for a new messaging channel. Use gateway-native ingress only

- for a non-channel surface that needs the Gateway runtime.

- - Submit `%LemonCore.RunRequest{}` through `LemonCore.RouterBridge.submit_run/1`.

- - Treat `{:error, :outcome_unknown}` as an ambiguous side effect, not a

- rejection. Webhook ingress acknowledges the HTTP delivery with a `200`

- `status: "outcome_unknown"` receipt and `retry_safe: false`, keeps the fixed

- run ID for reconciliation, and never turns that result into a retryable 5xx.

- Reserve idempotency keys atomically before submission. Reservation, replay

- read, or receipt-write failures fail closed with a retryable 503 without

- reporting acceptance. An unresolved reservation returns

- `status: "reservation_pending"` with `retry_safe: true`; a submitted sync run

- whose exact response has no durable receipt returns

- `status: "response_persistence_unknown"` with `retry_safe: false` rather than

- replaying a generic accepted response. Mutation exceptions and exits are

- outcome-unknown, never definite rejection. Hash the caller's idempotency key

- with its integration identity before building any durable key, receipt, or

- router request, and reject non-binary or blank payload keys before hashing.

- On upgrade, migrate legacy raw-key reservations into the hashed namespace by

- durably creating the hashed fence and exact response first, then conditionally

- deleting the raw records; concurrent migration and any partial failure remain

- fail closed. A released-format `pending` row without a run ID remains a

- permanent ambiguous fence rather than manufacturing a replay identity; its

- bounded duplicate receipt is non-retryable and never claims acceptance.

- Expiry uses strict backend listing and removes the response

- snapshot before the primary execution fence on ordered backends; SQLite

- performs durable pairs in one transaction, and cleanup failures neither

- authorize replay nor advance the sweep watermark.

- - Build stable, unique session keys.

- - Return `:ignore` from `start_link/1` when disabled.

- - Resolve binding cwd and agent metadata through `BindingResolver`.

- ## Operations and Debugging

- Useful runtime inspection:

- Registry.lookup(LemonGateway.RunRegistry, "run_uuid")

- LemonGateway.Executor.validate_configured()

- LemonGateway.Config.get(:max_concurrent_runs)

- For stuck runs, inspect scheduler in-flight slots and `EngineLock` waiters. For an

- unready runtime, inspect `LemonGateway.Executor.validate_configured/0` and the

- configured `:executor` module; do not look for a registry entry or a selected engine.

- For auto-resume, inspect `LemonCore.ChatStateStore` and the native executor's resume

- state. For transport startup, inspect `TransportRegistry`; Telegram, Discord, XMTP,

- and email remain channel-owned.

-

- ### Test application isolation

-

- Umbrella tests share one BEAM and one OTP application controller. A test module that

- stops globally named runtime applications must capture which applications were

- running and restore that exact running set in `setup_all` cleanup. Limit this

- isolation to applications the unit test actually owns: gateway transport tests may

- restart `:lemon_gateway` and `:lemon_channels`, but must not stop

- `:lemon_control_plane` or other consumer applications. Each app suite establishes

- its own production baseline. Use `Application.ensure_all_started/1` for restoration

- so transitive runtime dependencies return through their production supervision

- trees.

-

- ## Integration Points

-

- 1. Submit execution through `LemonGateway.Runtime.submit_execution/1`.

- 2. Set `meta.notify_pid` to receive `{:lemon_gateway_run_completed, execution_request, completed}`.

- 3. Subscribe to `LemonCore.Bus` topic `"run:<run_id>"` for real-time run events.

- 4. Cancel through `LemonGateway.Runtime.cancel_by_run_id/2`.

- 5. Let `LemonCore.ChatStateStore` retain native and task resume state.

- ## Dependencies

- - `coding_agent` provides the configured native executor and its in-process tools.

- - `lemon_core` provides execution contracts, Store, Bus, Telemetry, resume formats,

- bindings, secrets, and canonical gateway configuration.

- - `lemon_channels` owns Telegram/Discord/XMTP channel adapters and consumes gateway

- bus events; LemonGateway does not depend on it directly.