AGENTS.md@apps/lemon_mcp · git:20260830.88ea693 · 2026-08-30 · sha256 ba5435d547e39d4a
AGENTS.md@apps/lemon_mcp git:20260830.88ea693A
Immutable. This exact content never changes and is served at /api/v1/blob/ba5435d547e39d4a.
# LemonMCP App - Agent Guidelines
## Purpose and Responsibilities
The `lemon_mcp` app implements both sides of the **Model Context Protocol (MCP)**, an open JSON-RPC 2.0 protocol that lets LLM applications discover and call external tools.
### Key Responsibilities
- **MCP Client**: Spawn external MCP servers as subprocesses (via stdio) or connect to Streamable HTTP / legacy HTTP+SSE MCP endpoints, perform the handshake, handle OAuth client-credentials, refresh-token, and PKCE authorization-code callback flows for Streamable HTTP, and proxy tool calls to them — enabling Lemon to consume third-party tool servers (e.g. `@modelcontextprotocol/server-filesystem`)
- **MCP Server**: Accept inbound MCP connections over HTTP and expose Lemon's own `CodingAgent` tools to external MCP clients
- **Protocol Layer**: Encode/decode JSON-RPC 2.0 messages, validate lifecycle ordering, and map between Lemon's internal tool types and the MCP wire format
Protocol version targeted: **"2024-11-05"**
## Architecture Overview
```
CLIENT SIDE SERVER SIDE
___________________ __________________________
| | | LemonMCP.Transport.HTTP |
| LemonMCP.Client | | unnamed :one_for_all sup |
| LemonMCP.Client.HTTP |__________________________|
| (GenServer) | | |
|___________________| v v
| Bandit / Plug LemonMCP.Server
v | (GenServer)
LemonMCP.Transport.Stdio v |
(GenServer, Port-based) LemonMCP.Server.Handler <+
| (pure request dispatcher)
v |
External MCP server v
process via stdin/stdout LemonMCP.ToolAdapter
- bridges to CodingAgent tools
- converts parameter schemas
```
Both sides share `LemonMCP.Protocol` for all struct definitions and encode/decode helpers.
## File Structure
```
lib/
lemon_mcp.ex # Public façade: protocol_version/0, parse_request/1,
# create_response/2, create_error/3-4, encode!/1
lemon_mcp/
protocol.ex # All MCP message structs + encode/decode + builder helpers
client.ex # GenServer client (stdio transport, state machine)
client/
http.ex # GenServer client (Streamable HTTP transport)
sse.ex # GenServer client (legacy HTTP+SSE transport)
server.ex # GenServer server (tool registry, init tracking)
tool_adapter.ex # CodingAgent → MCP bridge; also a __using__ macro
server/
handler.ex # Pure request router: initialize/initialized/tools/list/tools/call
transport/
stdio.ex # GenServer wrapping an Erlang Port for subprocess I/O
http.ex # Plug.Router + public HTTP transport facade
http/supervisor.ex # Owns the MCP Server + Bandit as one lifecycle
test/
lemon_mcp/
protocol_test.exs
client_test.exs
server_test.exs
tool_adapter_test.exs
server/
handler_test.exs
transport/
http_test.exs
stdio_test.exs
```
## Module Reference
| Module | Role |
|--------|------|
| `LemonMCP` | Thin public API; delegates to `Protocol` |
| `LemonMCP.Protocol` | All structs and encode/decode; also server builder helpers (`parse_request/1`, `create_response/2`, `create_error_response/4`, `error_code/1`, `server_capabilities/1`, `initialize_result/4`) |
| `LemonMCP.Client` | GenServer; owns stdio transport lifecycle, request correlation, timeout handling, resource/prompt calls, and optional `sampling/createMessage` callbacks or policies. It only advertises the MCP sampling capability when a `sampling_handler` or `sampling_policy` is configured. |
| `LemonMCP.Sampling` | Redacted policy wrapper for stdio `sampling/createMessage`; summarizes requests without raw prompt text, enforces max-token/model allowlists, and supports reviewed model-backed delegates. |
| `LemonMCP.Client.HTTP` | GenServer; owns Streamable HTTP initialize/initialized handshake, JSON or per-request SSE responses, session/protocol headers, OAuth protected-resource and authorization-server metadata discovery, optional OAuth client-credentials token acquisition with form-post or HTTP Basic client authentication, refresh-token grant retry when a token response supplies a refresh token, authorization-code PKCE callback/token exchange for public clients, storage-agnostic OAuth token cache load/save hooks, one-shot bearer reacquisition after later 401 challenges, synchronous request calls, timeout handling, and server capability metadata |
| `LemonMCP.Client.SSE` | GenServer; owns legacy HTTP+SSE endpoint discovery, POST/message response correlation, timeout handling, and server capability metadata |
| `LemonMCP.Transport.Stdio` | GenServer backed by an Erlang `Port`; newline-delimited JSON framing; calls `message_handler` callback for each inbound line |
| `LemonMCP.Server` | GenServer; holds tool list or `tool_provider` module reference; tracks `initialized` flag; also declares the `@behaviour` for tool provider modules |
| `LemonMCP.Server.Handler` | Pure functions; routes `JSONRPCRequest` structs to handlers; validates params; calls into `Server` GenServer |
| `LemonMCP.Transport.HTTP` | `Plug.Router` plus transport facade; an unnamed internal `:one_for_all` supervisor owns the MCP Server and Bandit listener, resolves the current server child for each request, supports single and batch JSON-RPC, and exposes `POST /mcp` plus `GET /health` |
| `LemonMCP.ToolAdapter` | Struct + functions that bridge `CodingAgent.Tools.*` modules to MCP format; also a `__using__` macro that generates a `@behaviour LemonMCP.Server` implementation at compile time |
## Key Types
### Protocol Structs (all in `LemonMCP.Protocol`)
```elixir
# Lifecycle
%Protocol.InitializeRequest{} # client → server: jsonrpc, id, method, params
%Protocol.InitializeResponse{} # server → client: jsonrpc, id, result | error
%Protocol.InitializedNotification{} # client → server: no id (notification)
# Tool operations
%Protocol.ToolListRequest{} # tools/list request
%Protocol.ToolListResponse{} # result: %{tools: [...]}
%Protocol.ToolCallRequest{} # params: %{name: String, arguments: map()}
%Protocol.ToolCallResponse{} # result: %{content: [...], isError: bool}
# Generic JSON-RPC
%Protocol.JSONRPCRequest{} # id, method, params — used by server routing
%Protocol.JSONRPCResponse{} # id, result — used by all server responses
%Protocol.JSONRPCError{} # code, message, data — embedded in responses
# Data types
%Protocol.Tool{} # name, description, inputSchema (JSON Schema map)
%Protocol.ToolCallResult{} # content: [%{type, text}], isError: boolean
```
**Content items** follow MCP spec: `%{type: "text", text: "..."}` is the common case; image and resource variants also exist.
### ToolAdapter Struct
```elixir
%LemonMCP.ToolAdapter{
cwd: String.t(), # working directory passed to CodingAgent tools
tool_opts: keyword(), # forwarded to each tool's tool/2 call
tool_modules: %{String.t() => module()} # name → CodingAgent.Tools.* module
}
```
### Client State Machine
The `LemonMCP.Client` and `LemonMCP.Client.HTTP` GenServers hold an internal `state` field (not to be confused with the GenServer state map):
```
:disconnected -> :initializing -> :ready
^-- sends InitializeRequest on init
^-- after InitializeResponse + InitializedNotification
```
Calls to `list_tools/2` and `call_tool/4` return `{:error, {:not_ready, current_state}}` unless in `:ready`.
## Common Modification Patterns
### Adding a New Tool to the ToolAdapter
The built-in tool map is a compile-time constant in `LemonMCP.ToolAdapter`:
```elixir
@builtin_tools %{
"my_tool" => CodingAgent.Tools.MyTool,
# ...existing entries...
}
```
The `ToolAdapter` calls `module.tool(cwd, tool_opts)` to get the tool struct, then converts its `parameters` list to a JSON Schema `inputSchema`. Parameter type atoms (`:string`, `:integer`, `:boolean`, etc.) are mapped via `map_parameter_type/1`.
### Implementing a Custom Tool Provider (Server Side)
Two options:
**Option A — `__using__` macro (compile-time adapter):**
```elixir
defmodule MyApp.MCPProvider do
use LemonMCP.ToolAdapter, cwd: "/path/to/project", exclude_tools: ["bash"]
end
```
This generates `list_tools/0` and `call_tool/2` backed by `ToolAdapter`.
**Option B — manual `@behaviour LemonMCP.Server`:**
```elixir
defmodule MyApp.MCPProvider do
@behaviour LemonMCP.Server
@impl true
def list_tools do
[%LemonMCP.Protocol.Tool{name: "my_tool", description: "...", inputSchema: %{...}}]
end
@impl true
def call_tool("my_tool", args) do
{:ok, %LemonMCP.Protocol.ToolCallResult{content: [%{type: "text", text: "done"}], isError: false}}
end
def call_tool(_, _), do: {:error, :unknown_tool}
end
```
Then start the server with `tool_provider: MyApp.MCPProvider`.
### Adding a New MCP Method to the Server
1. Add a match arm in `LemonMCP.Server.Handler.handle_request/2`
2. Implement a `handle_<method>/2` function in `Handler`
3. Add any persistent state needed to `LemonMCP.Server` and a `handle_call` clause
4. Add the capability to `Protocol.server_capabilities/1` if it's a new capability type
### Starting the HTTP Transport
```elixir
{:ok, _pid} = LemonMCP.Transport.HTTP.start_link(
port: 4048,
server_name: "My Server",
server_version: "1.0.0",
tool_provider: MyApp.MCPProvider
)
```
The HTTP transport starts an unnamed internal `:one_for_all` supervisor that
owns its `LemonMCP.Server` and Bandit listener. If either child fails, both are
replaced together, so the listener never retains a dead server and a restarted
listener never leaves the old server behind. `get_server_pid/0` follows the
most recently started live transport supervisor to its current server child,
falling back to older live instances rather than caching a child PID. Use
`get_server_pid/1` with the supervisor returned by `start_link/1` when multiple
transports are running.
**Configuration alternative** (app config):
```elixir
config :lemon_mcp, :http_transport,
enabled: true,
port: 4048,
ip: {127, 0, 0, 1},
server_name: "Lemon MCP Server",
server_version: "0.1.0"
```
### Connecting a Client to an External MCP Server
Stdio client:
```elixir
{:ok, client} = LemonMCP.Client.start_link(
command: "npx",
args: ["-y", "@modelcontextprotocol/server-filesystem", "/path"],
env: [{"HOME", System.get_env("HOME")}],
timeout_ms: 15_000
)
# Wait for :ready (handshake is async during init)
{:ok, tools} = LemonMCP.Client.list_tools(client)
{:ok, content} = LemonMCP.Client.call_tool(client, "read_file", %{"path" => "/tmp/test.txt"})
# content is a list of %{type: "text", text: "..."} maps
```
Stdio sampling is opt-in at the low-level client boundary. Prefer
`sampling_policy` for model-backed sampling so redacted review and local limits
run before a delegate sees the raw request:
```elixir
{:ok, client} = LemonMCP.Client.start_link(
command: "node",
args: ["server.js"],
sampling_policy: [
mode: :reviewed_model,
reviewer: fn summary -> :approve end,
delegate: fn params, summary ->
{:ok, model_sampling_result(params, summary)}
end,
max_tokens: 1_024,
allowed_models: ["lemon"]
]
)
```
Use `sampling_handler` only for low-level integrations that already own review
and policy checks:
```elixir
{:ok, client} = LemonMCP.Client.start_link(
command: "node",
args: ["server.js"],
sampling_handler: fn params ->
{:ok,
%{
"role" => "assistant",
"content" => %{"type" => "text", "text" => "sampled"},
"model" => "lemon",
"stopReason" => "endTurn"
}}
end
)
```
Do not advertise sampling by adding `"sampling"` to static capabilities unless
the client can actually answer server sampling requests. Prefer the
`sampling_policy` or `sampling_handler` option so capability advertisement and
response routing stay tied together.
When a stdio server is configured through `LemonSkills.McpSource`, the source
layer may use `reviewer: :ops_approval` to bridge reviewed sampling into
`LemonCore.ExecApprovals`. That production-facing bridge stores only redacted
`mcp_<server>_sampling` summaries for approval surfaces before the low-level
MCP delegate sees raw params. Approval summaries include the safe request hash,
model, token, message, role, and content-kind metadata.
Streamable HTTP client:
```elixir
{:ok, client} = LemonMCP.Client.HTTP.start_link(
url: "http://localhost:3000/mcp",
headers: [{"Authorization", "Bearer token"}]
)
{:ok, tools} = LemonMCP.Client.HTTP.list_tools(client)
{:ok, content} = LemonMCP.Client.HTTP.call_tool(client, "search", %{"query" => "beam"})
```
OAuth-protected Streamable HTTP servers can use configured client credentials:
```elixir
{:ok, client} = LemonMCP.Client.HTTP.start_link(
url: "https://example.com/mcp",
oauth: [client_id: "client", client_secret: "secret", scopes: ["tools"]]
)
```
The client uses `WWW-Authenticate` protected-resource metadata and declared
authorization-server metadata to find the token endpoint, requests a bearer
token with `grant_type=client_credentials`, and retries the protected MCP
request. If a later 401 arrives and the previous token response supplied a
refresh token, the client prefers `grant_type=refresh_token`, rotates the stored
refresh token when the token endpoint returns a replacement, and only falls back
to client credentials when no refresh token is available or refresh fails.
Metadata discovery requests use metadata-only headers; configured MCP headers
and acquired bearer tokens are not forwarded to metadata URLs.
Public-client authorization-code + PKCE flows use an
`authorization_code_provider` callback as the narrow UI boundary. The callback
receives the authorization URL, state, verifier/challenge metadata, resource,
scope, and redirect URI, then returns a code plus matching state for token
exchange. OAuth tokens remain process-local unless the caller supplies
`oauth_token_cache: [load: ..., save: ...]` or the compatibility
`oauth_token_loader` / `oauth_token_persister` callbacks. The client loads a
cached access/refresh token before initialize and persists successful
client-credentials, refresh-token, and PKCE token responses, including refresh
token rotation. `lemon_mcp` deliberately does not own secret storage, loopback
HTTP callback hosting, or operator UI; those belong in higher-level Lemon
runtime or skills surfaces. If a higher-level provider may block for an
interactive authorization flow, pass the GenServer `:timeout` start option
alongside the client config.
HTTP resource/prompt calls use the same API shape and degrade with MCP RPC
errors when the remote endpoint does not implement those methods.
Legacy HTTP+SSE client:
```elixir
{:ok, client} = LemonMCP.Client.SSE.start_link(
url: "http://localhost:3000/sse",
headers: [{"Authorization", "Bearer token"}]
)
{:ok, tools} = LemonMCP.Client.SSE.list_tools(client)
{:ok, content} = LemonMCP.Client.SSE.call_tool(client, "search", %{"query" => "beam"})
```
## Error Handling Distinctions
The `ToolAdapter` distinguishes two categories of tool failure:
| Failure type | Return value | MCP representation |
|---|---|---|
| Tool-declared error (`{:error, reason}` from `execute/4`) | `{:ok, %ToolCallResult{isError: true}}` | Successful tool call with `isError: true` in content |
| Adapter/tool crash (exception during `execute/4`) | `{:error, {:tool_crash, message}}` | Propagates as a server-level error (not a tool result) |
This matters because the `Handler` treats `{:error, reason}` from `Server.call_tool/3` as a JSON-RPC error response (`-32600`), whereas an `{:ok, %ToolCallResult{isError: true}}` becomes a successful protocol response with `isError` set.
## Error Codes Reference
Standard JSON-RPC 2.0 codes used throughout:
| Atom | Code | Meaning |
|------|------|---------|
| `:parse_error` | -32700 | Invalid JSON |
| `:invalid_request` | -32600 | Request structure wrong |
| `:method_not_found` | -32601 | Unknown method / unknown tool |
| `:invalid_params` | -32602 | Missing or malformed params |
| `:internal_error` | -32603 | Unexpected server failure |
| `:server_not_initialized` | -32002 | `tools/list` or `tools/call` before `initialized` |
Use `Protocol.error_code(:atom)` to convert; falls back to `-32603` for unknown atoms.
## Lifecycle and Release Composition
`lemon_mcp` is a library application. It has no application callback or
supervisor and starts no processes by default. `LemonMCP.Client`,
`LemonMCP.Server`, and the transport modules are started on demand; the host
application that needs one owns its supervision strategy.
The `lemon_runtime_min` and `lemon_runtime_full` releases explicitly include
`lemon_mcp: :load` before `coding_agent` and `lemon_skills`. This assembles and
loads the client modules so `LemonSkills.McpSource.mcp_enabled?/0` can discover
them, without pretending the MCP library owns long-lived runtime processes.
`release-smoke.yml` builds both profiles and evaluates that contract inside
each packaged release; `scripts/verify_release_runtime_boot` repeats it against
each min/full release tarball. The client and source modules must load, MCP
feature detection must be enabled, `:lemon_mcp` must be loaded but not started,
and no application callback or `LemonMCP.Application` module may exist.
## Testing Guidance
### Running Tests
```bash
# All lemon_mcp tests (from umbrella root)
mix test apps/lemon_mcp/test
# Specific file
mix test apps/lemon_mcp/test/lemon_mcp/server/handler_test.exs
```
### Test Patterns in Use
**Server tests** (`server_test.exs`, `handler_test.exs`) start real `LemonMCP.Server` GenServers in `setup` with inline tool lists and a `tool_handler` function. No mocking needed — the server is pure in-process state.
**ToolAdapter tests** (`tool_adapter_test.exs`) define minimal tool modules inline using `AgentTool` / `AgentToolResult` structs, then build `%ToolAdapter{}` structs directly (bypassing `new/2`). This avoids needing real `CodingAgent` tool modules:
```elixir
defmodule MyFakeTool do
def tool(_cwd, _opts) do
%LemonAgent.Types.AgentTool{
name: "my_tool",
description: "...",
parameters: [],
label: "My Tool",
execute: fn _id, _params, _signal, _on_update -> "some result" end
}
end
end
adapter = %LemonMCP.ToolAdapter{
cwd: "/tmp",
tool_opts: [],
tool_modules: %{"my_tool" => MyFakeTool}
}
```
**Stdio transport tests** (`transport/stdio_test.exs`) require an actual executable to be present. Tests that spawn processes should be tagged or skipped in CI environments without the target command available.
**Client tests** (`client_test.exs`) similarly depend on spawning real processes; they are integration-level.
### Testing a New Tool Provider Module
```elixir
defmodule MyProviderTest do
use ExUnit.Case, async: true
alias LemonMCP.{Protocol, Server, Server.Handler}
setup do
{:ok, server} = Server.start_link(tool_provider: MyApp.MCPProvider)
# Must mark initialized before tools/list or tools/call will work
Server.mark_initialized(server)
%{server: server}
end
test "lists my tools", %{server: server} do
tools = Server.list_tools(server)
assert Enum.any?(tools, &(&1.name == "my_tool"))
end
test "calls my tool via handler", %{server: server} do
request = %Protocol.JSONRPCRequest{
id: "1", method: "tools/call",
params: %{"name" => "my_tool", "arguments" => %{}}
}
response = Handler.handle_request(request, server)
assert response.result[:isError] == false
end
end
```
## Gotchas
- **Server must be initialized before tool operations.** `tools/list` and `tools/call` both return `-32002 server_not_initialized` until `Handler.handle_initialized/2` has been called (which calls `Server.mark_initialized/1`). In tests, call `Server.mark_initialized(server)` explicitly in setup.
- **`create_error/4` vs `create_error_response/4`**: `Protocol.create_error/4` embeds the error inside `result` (a quirk); `Protocol.create_error_response/4` puts it at the top level as `error:` per the JSON-RPC spec. The `Handler` uses `create_error_response/4` for proper wire format. Prefer `create_error_response/4` in server-side code.
- **Protocol version check is exact-match only.** `Handler.handle_initialize/2` calls `compatible_version?/1` which checks against a `supported_versions/0` list. Currently only `"2024-11-05"` is accepted. A client sending a different version string will get an `:invalid_request` error.
- **Stdio transport stderr merges into stdout.** `Port.open/2` is started with `:stderr_to_stdout`, so stderr from the MCP server subprocess arrives as ordinary data lines. This can cause spurious JSON parse warnings if the subprocess logs to stderr on startup (e.g. `npx` install output).
- **ToolAdapter `new/2` options use `include_tools`/`exclude_tools` but the `__using__` macro does not pass `include_tools`.** If you need to restrict tools via the macro, pass `:exclude_tools` explicitly.
- **HTTP transport port defaults to `0` in test env.** `@default_port` is set at compile time via `Mix.env()`. In test, a random available port is assigned, which avoids conflicts. The PID returned by `start_link/1` is the unnamed transport supervisor, not the Bandit listener. Use `get_server_pid/1` for a specific transport or the compatibility `get_server_pid/0` lookup for the newest live instance.
- **There is no `LemonMCP.Application` callback.** All client, server, and
transport processes must be started and supervised by the consuming app.
Keep the runtime release entry at `:load`; changing it to `:permanent` does
not create useful lifecycle ownership.
- **`ToolAdapter` calls `module.tool/2` during both `list_tools/1` and `call_tool/3`.** The tool definition is fetched twice per operation. If constructing the `AgentTool` struct is expensive, this could matter.
## How This App Connects to Other Umbrella Apps
- **`coding_agent`** (dependency): `ToolAdapter` maps `CodingAgent.Tools.*` module names to MCP tool definitions. All `@builtin_tools` entries reference `CodingAgent.Tools` modules.
- **`agent_core`** (dependency): `LemonAgent.Types.AgentToolResult` and `LemonAgent.Types.AgentTool` are used by `ToolAdapter` for result conversion.
- **Consumers**: Apps with a direct compile-time dependency use
`{:lemon_mcp, in_umbrella: true}` and call `LemonMCP.Client.start_link/1`,
`LemonMCP.Client.HTTP.start_link/1`, or
`LemonMCP.Transport.HTTP.start_link/1`. `LemonSkills.McpSource` deliberately
discovers the client dynamically; the runtime release contract assembles the
library for that path.