AGENTS.md@apps/lemon_mcp git:20260830.97a47f1A

Immutable. This exact content never changes and is served at /api/v1/blob/707a14d971e4d946.

# LemonMCP App - Agent Guidelines

## Purpose and Responsibilities

The `lemon_mcp` app implements both sides of the **Model Context Protocol (MCP)**, an open JSON-RPC 2.0 protocol that lets LLM applications discover and call external tools.

### Key Responsibilities

- **MCP Client**: Spawn external MCP servers as subprocesses (via stdio) or connect to Streamable HTTP / legacy HTTP+SSE MCP endpoints, perform the handshake, handle OAuth client-credentials, refresh-token, and PKCE authorization-code callback flows for Streamable HTTP, and proxy tool calls to them — enabling Lemon to consume third-party tool servers (e.g. `@modelcontextprotocol/server-filesystem`)
- **MCP Server**: Accept inbound MCP connections over HTTP and expose Lemon's own `CodingAgent` tools to external MCP clients
- **Protocol Layer**: Encode/decode JSON-RPC 2.0 messages, validate lifecycle ordering, and map between Lemon's internal tool types and the MCP wire format

Protocol version targeted: **"2024-11-05"**

## Architecture Overview

```
                     CLIENT SIDE                           SERVER SIDE
                 ___________________               __________________________
                |                   |             | LemonMCP.Transport.HTTP  |
                |  LemonMCP.Client  |             | unnamed :one_for_all sup |
                | LemonMCP.Client.HTTP            |__________________________|
                |    (GenServer)    |                    |             |
                |___________________|                    v             v
                         |                        Bandit / Plug   LemonMCP.Server
                         v                              |          (GenServer)
              LemonMCP.Transport.Stdio                 v             |
               (GenServer, Port-based)       LemonMCP.Server.Handler <+
                         |                    (pure request dispatcher)
                         v                                    |
               External MCP server                        v
               process via stdin/stdout          LemonMCP.ToolAdapter
                                                 - bridges to CodingAgent tools
                                                 - converts parameter schemas
```

Both sides share `LemonMCP.Protocol` for all struct definitions and encode/decode helpers.

## File Structure

```
lib/
  lemon_mcp.ex                     # Public façade: protocol_version/0, parse_request/1,
                                   #   create_response/2, create_error/3-4, encode!/1
  lemon_mcp/
    protocol.ex                    # All MCP message structs + encode/decode + builder helpers
    client.ex                      # GenServer client (stdio transport, state machine)
    client/
      http.ex                      # GenServer client (Streamable HTTP transport)
      sse.ex                       # GenServer client (legacy HTTP+SSE transport)
    server.ex                      # GenServer server (tool registry, init tracking)
    tool_adapter.ex                # CodingAgent → MCP bridge; also a __using__ macro
    server/
      handler.ex                   # Pure request router: initialize/initialized/tools/list/tools/call
    transport/
      stdio.ex                     # GenServer wrapping an Erlang Port for subprocess I/O
      http.ex                      # Plug.Router + public HTTP transport facade
      http/supervisor.ex           # Owns the MCP Server + Bandit as one lifecycle

test/
  lemon_mcp/
    protocol_test.exs
    client_test.exs
    server_test.exs
    tool_adapter_test.exs
    server/
      handler_test.exs
    transport/
      http_test.exs
      stdio_test.exs
```

## Module Reference

| Module | Role |
|--------|------|
| `LemonMCP` | Thin public API; delegates to `Protocol` |
| `LemonMCP.Protocol` | All structs and encode/decode; also server builder helpers (`parse_request/1`, `create_response/2`, `create_error_response/4`, `error_code/1`, `server_capabilities/1`, `initialize_result/4`) |
| `LemonMCP.Client` | GenServer; owns stdio transport lifecycle, request correlation, timeout handling, resource/prompt calls, and optional `sampling/createMessage` callbacks or policies. It only advertises the MCP sampling capability when a `sampling_handler` or `sampling_policy` is configured. |
| `LemonMCP.Sampling` | Redacted policy wrapper for stdio `sampling/createMessage`; summarizes requests without raw prompt text, enforces max-token/model allowlists, and supports reviewed model-backed delegates. |
| `LemonMCP.Client.HTTP` | GenServer; owns Streamable HTTP initialize/initialized handshake, JSON or per-request SSE responses, session/protocol headers, OAuth protected-resource and authorization-server metadata discovery, optional OAuth client-credentials token acquisition with form-post or HTTP Basic client authentication, refresh-token grant retry when a token response supplies a refresh token, authorization-code PKCE callback/token exchange for public clients, storage-agnostic OAuth token cache load/save hooks, one-shot bearer reacquisition after later 401 challenges, synchronous request calls, timeout handling, and server capability metadata |
| `LemonMCP.Client.SSE` | GenServer; owns legacy HTTP+SSE endpoint discovery, POST/message response correlation, timeout handling, and server capability metadata |
| `LemonMCP.Transport.Stdio` | GenServer backed by an Erlang `Port`; newline-delimited JSON framing; calls `message_handler` callback for each inbound line |
| `LemonMCP.Server` | GenServer; holds tool list or `tool_provider` module reference; tracks `initialized` flag; also declares the `@behaviour` for tool provider modules |
| `LemonMCP.Server.Handler` | Pure functions; routes `JSONRPCRequest` structs to handlers; validates params; calls into `Server` GenServer |
| `LemonMCP.Transport.HTTP` | `Plug.Router` plus transport facade; an unnamed internal `:one_for_all` supervisor owns the MCP Server and Bandit listener, resolves the current server child for each request, supports single and batch JSON-RPC, and exposes `POST /mcp` plus `GET /health` |
| `LemonMCP.ToolAdapter` | Struct + functions that bridge `CodingAgent.Tools.*` modules to MCP format; also a `__using__` macro that generates a `@behaviour LemonMCP.Server` implementation at compile time |

## Key Types

### Protocol Structs (all in `LemonMCP.Protocol`)

```elixir
# Lifecycle
%Protocol.InitializeRequest{}   # client → server: jsonrpc, id, method, params
%Protocol.InitializeResponse{}  # server → client: jsonrpc, id, result | error
%Protocol.InitializedNotification{}  # client → server: no id (notification)

# Tool operations
%Protocol.ToolListRequest{}     # tools/list request
%Protocol.ToolListResponse{}    # result: %{tools: [...]}
%Protocol.ToolCallRequest{}     # params: %{name: String, arguments: map()}
%Protocol.ToolCallResponse{}    # result: %{content: [...], isError: bool}

# Generic JSON-RPC
%Protocol.JSONRPCRequest{}      # id, method, params — used by server routing
%Protocol.JSONRPCResponse{}     # id, result — used by all server responses
%Protocol.JSONRPCError{}        # code, message, data — embedded in responses

# Data types
%Protocol.Tool{}                # name, description, inputSchema (JSON Schema map)
%Protocol.ToolCallResult{}      # content: [%{type, text}], isError: boolean
```

**Content items** follow MCP spec: `%{type: "text", text: "..."}` is the common case; image and resource variants also exist.

### ToolAdapter Struct

```elixir
%LemonMCP.ToolAdapter{
  cwd: String.t(),               # working directory passed to CodingAgent tools
  tool_opts: keyword(),          # forwarded to each tool's tool/2 call
  tool_modules: %{String.t() => module()}  # name → CodingAgent.Tools.* module
}
```

### Client State Machine

The `LemonMCP.Client` and `LemonMCP.Client.HTTP` GenServers hold an internal `state` field (not to be confused with the GenServer state map):

```
:disconnected  ->  :initializing  ->  :ready
                        ^-- sends InitializeRequest on init
                                       ^-- after InitializeResponse + InitializedNotification
```

Calls to `list_tools/2` and `call_tool/4` return `{:error, {:not_ready, current_state}}` unless in `:ready`.

## Common Modification Patterns

### Adding a New Tool to the ToolAdapter

The built-in tool map is a compile-time constant in `LemonMCP.ToolAdapter`:

```elixir
@builtin_tools %{
  "my_tool" => CodingAgent.Tools.MyTool,
  # ...existing entries...
}
```

The `ToolAdapter` calls `module.tool(cwd, tool_opts)` to get the tool struct, then converts its `parameters` list to a JSON Schema `inputSchema`. Parameter type atoms (`:string`, `:integer`, `:boolean`, etc.) are mapped via `map_parameter_type/1`.

### Implementing a Custom Tool Provider (Server Side)

Two options:

**Option A — `__using__` macro (compile-time adapter):**

```elixir
defmodule MyApp.MCPProvider do
  use LemonMCP.ToolAdapter, cwd: "/path/to/project", exclude_tools: ["bash"]
end
```

This generates `list_tools/0` and `call_tool/2` backed by `ToolAdapter`.

**Option B — manual `@behaviour LemonMCP.Server`:**

```elixir
defmodule MyApp.MCPProvider do
  @behaviour LemonMCP.Server

  @impl true
  def list_tools do
    [%LemonMCP.Protocol.Tool{name: "my_tool", description: "...", inputSchema: %{...}}]
  end

  @impl true
  def call_tool("my_tool", args) do
    {:ok, %LemonMCP.Protocol.ToolCallResult{content: [%{type: "text", text: "done"}], isError: false}}
  end

  def call_tool(_, _), do: {:error, :unknown_tool}
end
```

Then start the server with `tool_provider: MyApp.MCPProvider`.

### Adding a New MCP Method to the Server

1. Add a match arm in `LemonMCP.Server.Handler.handle_request/2`
2. Implement a `handle_<method>/2` function in `Handler`
3. Add any persistent state needed to `LemonMCP.Server` and a `handle_call` clause
4. Add the capability to `Protocol.server_capabilities/1` if it's a new capability type

### Starting the HTTP Transport

```elixir
{:ok, _pid} = LemonMCP.Transport.HTTP.start_link(
  port: 4048,
  server_name: "My Server",
  server_version: "1.0.0",
  tool_provider: MyApp.MCPProvider
)
```

The HTTP transport starts an unnamed internal `:one_for_all` supervisor that
owns its `LemonMCP.Server` and Bandit listener. If either child fails, both are
replaced together, so the listener never retains a dead server and a restarted
listener never leaves the old server behind. `get_server_pid/0` follows the
most recently started transport supervisor to its current server child rather
than caching a child PID.

**Configuration alternative** (app config):

```elixir
config :lemon_mcp, :http_transport,
  enabled: true,
  port: 4048,
  ip: {127, 0, 0, 1},
  server_name: "Lemon MCP Server",
  server_version: "0.1.0"
```

### Connecting a Client to an External MCP Server

Stdio client:

```elixir
{:ok, client} = LemonMCP.Client.start_link(
  command: "npx",
  args: ["-y", "@modelcontextprotocol/server-filesystem", "/path"],
  env: [{"HOME", System.get_env("HOME")}],
  timeout_ms: 15_000
)

# Wait for :ready (handshake is async during init)
{:ok, tools} = LemonMCP.Client.list_tools(client)
{:ok, content} = LemonMCP.Client.call_tool(client, "read_file", %{"path" => "/tmp/test.txt"})
# content is a list of %{type: "text", text: "..."} maps
```

Stdio sampling is opt-in at the low-level client boundary. Prefer
`sampling_policy` for model-backed sampling so redacted review and local limits
run before a delegate sees the raw request:

```elixir
{:ok, client} = LemonMCP.Client.start_link(
  command: "node",
  args: ["server.js"],
  sampling_policy: [
    mode: :reviewed_model,
    reviewer: fn summary -> :approve end,
    delegate: fn params, summary ->
      {:ok, model_sampling_result(params, summary)}
    end,
    max_tokens: 1_024,
    allowed_models: ["lemon"]
  ]
)
```

Use `sampling_handler` only for low-level integrations that already own review
and policy checks:

```elixir
{:ok, client} = LemonMCP.Client.start_link(
  command: "node",
  args: ["server.js"],
  sampling_handler: fn params ->
    {:ok,
     %{
       "role" => "assistant",
       "content" => %{"type" => "text", "text" => "sampled"},
       "model" => "lemon",
       "stopReason" => "endTurn"
     }}
  end
)
```

Do not advertise sampling by adding `"sampling"` to static capabilities unless
the client can actually answer server sampling requests. Prefer the
`sampling_policy` or `sampling_handler` option so capability advertisement and
response routing stay tied together.

When a stdio server is configured through `LemonSkills.McpSource`, the source
layer may use `reviewer: :ops_approval` to bridge reviewed sampling into
`LemonCore.ExecApprovals`. That production-facing bridge stores only redacted
`mcp_<server>_sampling` summaries for approval surfaces before the low-level
MCP delegate sees raw params. Approval summaries include the safe request hash,
model, token, message, role, and content-kind metadata.

Streamable HTTP client:

```elixir
{:ok, client} = LemonMCP.Client.HTTP.start_link(
  url: "http://localhost:3000/mcp",
  headers: [{"Authorization", "Bearer token"}]
)

{:ok, tools} = LemonMCP.Client.HTTP.list_tools(client)
{:ok, content} = LemonMCP.Client.HTTP.call_tool(client, "search", %{"query" => "beam"})
```

OAuth-protected Streamable HTTP servers can use configured client credentials:

```elixir
{:ok, client} = LemonMCP.Client.HTTP.start_link(
  url: "https://example.com/mcp",
  oauth: [client_id: "client", client_secret: "secret", scopes: ["tools"]]
)
```

The client uses `WWW-Authenticate` protected-resource metadata and declared
authorization-server metadata to find the token endpoint, requests a bearer
token with `grant_type=client_credentials`, and retries the protected MCP
request. If a later 401 arrives and the previous token response supplied a
refresh token, the client prefers `grant_type=refresh_token`, rotates the stored
refresh token when the token endpoint returns a replacement, and only falls back
to client credentials when no refresh token is available or refresh fails.
Metadata discovery requests use metadata-only headers; configured MCP headers
and acquired bearer tokens are not forwarded to metadata URLs.

Public-client authorization-code + PKCE flows use an
`authorization_code_provider` callback as the narrow UI boundary. The callback
receives the authorization URL, state, verifier/challenge metadata, resource,
scope, and redirect URI, then returns a code plus matching state for token
exchange. OAuth tokens remain process-local unless the caller supplies
`oauth_token_cache: [load: ..., save: ...]` or the compatibility
`oauth_token_loader` / `oauth_token_persister` callbacks. The client loads a
cached access/refresh token before initialize and persists successful
client-credentials, refresh-token, and PKCE token responses, including refresh
token rotation. `lemon_mcp` deliberately does not own secret storage, loopback
HTTP callback hosting, or operator UI; those belong in higher-level Lemon
runtime or skills surfaces. If a higher-level provider may block for an
interactive authorization flow, pass the GenServer `:timeout` start option
alongside the client config.

HTTP resource/prompt calls use the same API shape and degrade with MCP RPC
errors when the remote endpoint does not implement those methods.

Legacy HTTP+SSE client:

```elixir
{:ok, client} = LemonMCP.Client.SSE.start_link(
  url: "http://localhost:3000/sse",
  headers: [{"Authorization", "Bearer token"}]
)

{:ok, tools} = LemonMCP.Client.SSE.list_tools(client)
{:ok, content} = LemonMCP.Client.SSE.call_tool(client, "search", %{"query" => "beam"})
```

## Error Handling Distinctions

The `ToolAdapter` distinguishes two categories of tool failure:

| Failure type | Return value | MCP representation |
|---|---|---|
| Tool-declared error (`{:error, reason}` from `execute/4`) | `{:ok, %ToolCallResult{isError: true}}` | Successful tool call with `isError: true` in content |
| Adapter/tool crash (exception during `execute/4`) | `{:error, {:tool_crash, message}}` | Propagates as a server-level error (not a tool result) |

This matters because the `Handler` treats `{:error, reason}` from `Server.call_tool/3` as a JSON-RPC error response (`-32600`), whereas an `{:ok, %ToolCallResult{isError: true}}` becomes a successful protocol response with `isError` set.

## Error Codes Reference

Standard JSON-RPC 2.0 codes used throughout:

| Atom | Code | Meaning |
|------|------|---------|
| `:parse_error` | -32700 | Invalid JSON |
| `:invalid_request` | -32600 | Request structure wrong |
| `:method_not_found` | -32601 | Unknown method / unknown tool |
| `:invalid_params` | -32602 | Missing or malformed params |
| `:internal_error` | -32603 | Unexpected server failure |
| `:server_not_initialized` | -32002 | `tools/list` or `tools/call` before `initialized` |

Use `Protocol.error_code(:atom)` to convert; falls back to `-32603` for unknown atoms.

## Lifecycle and Release Composition

`lemon_mcp` is a library application. It has no application callback or
supervisor and starts no processes by default. `LemonMCP.Client`,
`LemonMCP.Server`, and the transport modules are started on demand; the host
application that needs one owns its supervision strategy.

The `lemon_runtime_min` and `lemon_runtime_full` releases explicitly include
`lemon_mcp: :load` before `coding_agent` and `lemon_skills`. This assembles and
loads the client modules so `LemonSkills.McpSource.mcp_enabled?/0` can discover
them, without pretending the MCP library owns long-lived runtime processes.
`release-smoke.yml` builds both profiles and evaluates that contract inside
each packaged release; `scripts/verify_release_runtime_boot` repeats it against
each min/full release tarball. The client and source modules must load, MCP
feature detection must be enabled, `:lemon_mcp` must be loaded but not started,
and no application callback or `LemonMCP.Application` module may exist.

## Testing Guidance

### Running Tests

```bash
# All lemon_mcp tests (from umbrella root)
mix test apps/lemon_mcp/test

# Specific file
mix test apps/lemon_mcp/test/lemon_mcp/server/handler_test.exs
```

### Test Patterns in Use

**Server tests** (`server_test.exs`, `handler_test.exs`) start real `LemonMCP.Server` GenServers in `setup` with inline tool lists and a `tool_handler` function. No mocking needed — the server is pure in-process state.

**ToolAdapter tests** (`tool_adapter_test.exs`) define minimal tool modules inline using `AgentTool` / `AgentToolResult` structs, then build `%ToolAdapter{}` structs directly (bypassing `new/2`). This avoids needing real `CodingAgent` tool modules:

```elixir
defmodule MyFakeTool do
  def tool(_cwd, _opts) do
    %LemonAgent.Types.AgentTool{
      name: "my_tool",
      description: "...",
      parameters: [],
      label: "My Tool",
      execute: fn _id, _params, _signal, _on_update -> "some result" end
    }
  end
end

adapter = %LemonMCP.ToolAdapter{
  cwd: "/tmp",
  tool_opts: [],
  tool_modules: %{"my_tool" => MyFakeTool}
}
```

**Stdio transport tests** (`transport/stdio_test.exs`) require an actual executable to be present. Tests that spawn processes should be tagged or skipped in CI environments without the target command available.

**Client tests** (`client_test.exs`) similarly depend on spawning real processes; they are integration-level.

### Testing a New Tool Provider Module

```elixir
defmodule MyProviderTest do
  use ExUnit.Case, async: true

  alias LemonMCP.{Protocol, Server, Server.Handler}

  setup do
    {:ok, server} = Server.start_link(tool_provider: MyApp.MCPProvider)
    # Must mark initialized before tools/list or tools/call will work
    Server.mark_initialized(server)
    %{server: server}
  end

  test "lists my tools", %{server: server} do
    tools = Server.list_tools(server)
    assert Enum.any?(tools, &(&1.name == "my_tool"))
  end

  test "calls my tool via handler", %{server: server} do
    request = %Protocol.JSONRPCRequest{
      id: "1", method: "tools/call",
      params: %{"name" => "my_tool", "arguments" => %{}}
    }
    response = Handler.handle_request(request, server)
    assert response.result[:isError] == false
  end
end
```

## Gotchas

- **Server must be initialized before tool operations.** `tools/list` and `tools/call` both return `-32002 server_not_initialized` until `Handler.handle_initialized/2` has been called (which calls `Server.mark_initialized/1`). In tests, call `Server.mark_initialized(server)` explicitly in setup.

- **`create_error/4` vs `create_error_response/4`**: `Protocol.create_error/4` embeds the error inside `result` (a quirk); `Protocol.create_error_response/4` puts it at the top level as `error:` per the JSON-RPC spec. The `Handler` uses `create_error_response/4` for proper wire format. Prefer `create_error_response/4` in server-side code.

- **Protocol version check is exact-match only.** `Handler.handle_initialize/2` calls `compatible_version?/1` which checks against a `supported_versions/0` list. Currently only `"2024-11-05"` is accepted. A client sending a different version string will get an `:invalid_request` error.

- **Stdio transport stderr merges into stdout.** `Port.open/2` is started with `:stderr_to_stdout`, so stderr from the MCP server subprocess arrives as ordinary data lines. This can cause spurious JSON parse warnings if the subprocess logs to stderr on startup (e.g. `npx` install output).

- **ToolAdapter `new/2` options use `include_tools`/`exclude_tools` but the `__using__` macro does not pass `include_tools`.** If you need to restrict tools via the macro, pass `:exclude_tools` explicitly.

- **HTTP transport port defaults to `0` in test env.** `@default_port` is set at compile time via `Mix.env()`. In test, a random available port is assigned, which avoids conflicts. The PID returned by `start_link/1` is the unnamed transport supervisor, not the Bandit listener. Use `get_server_pid/0` for the current protocol server child.

- **There is no `LemonMCP.Application` callback.** All client, server, and
  transport processes must be started and supervised by the consuming app.
  Keep the runtime release entry at `:load`; changing it to `:permanent` does
  not create useful lifecycle ownership.

- **`ToolAdapter` calls `module.tool/2` during both `list_tools/1` and `call_tool/3`.** The tool definition is fetched twice per operation. If constructing the `AgentTool` struct is expensive, this could matter.

## How This App Connects to Other Umbrella Apps

- **`coding_agent`** (dependency): `ToolAdapter` maps `CodingAgent.Tools.*` module names to MCP tool definitions. All `@builtin_tools` entries reference `CodingAgent.Tools` modules.
- **`agent_core`** (dependency): `LemonAgent.Types.AgentToolResult` and `LemonAgent.Types.AgentTool` are used by `ToolAdapter` for result conversion.
- **Consumers**: Apps with a direct compile-time dependency use
  `{:lemon_mcp, in_umbrella: true}` and call `LemonMCP.Client.start_link/1`,
  `LemonMCP.Client.HTTP.start_link/1`, or
  `LemonMCP.Transport.HTTP.start_link/1`. `LemonSkills.McpSource` deliberately
  discovers the client dynamically; the runtime release contract assembles the
  library for that path.