git:20260927.572184c to git:20260927.979dcab

# LemonAutomation

Elixir app for cron jobs, heartbeats, and automation tasks in the Lemon umbrella.

## Purpose and Responsibilities

LemonAutomation provides scheduled and triggered automation for agents:

- **Cron Jobs** - Schedule agent prompts with cron expressions

- **Heartbeats** - Cron jobs flagged `meta.heartbeat`; exact `HEARTBEAT_OK` replies are suppressed, everything else is delivered

- **Delivery** - Finished run results go to an explicit `deliver_to` target, the originating chat, or the paired owner chat (`CronDelivery`)

- **Daily Brief** - Opt-in `[automation] daily_brief` heartbeat reconciled at boot

- **Wake** - Immediate manual triggering of scheduled jobs

- **Goal Continuation** - Supervised one-shot continuation runs for active session goals

- **Goal Loop Preview** - One supervised judge tick that continues, completes, or pauses a goal

- **Kanban Dispatch** - Supervised leasing/reclaim loop for durable board tasks

- **Portable Blueprints** - Preview and explicitly confirm profile-local skill bundles plus one agent cron job

- **Run Tracking** - Full lifecycle tracking of job executions

## Supervision Tree

```

LemonAutomation.Supervisor (one_for_one)

+-- LemonAutomation.TaskSupervisor (Task.Supervisor)

+-- LemonAutomation.CronManager (GenServer; runs Heartbeat policy + CronDelivery at terminalization)

+-- Task :daily_brief_sync (one-shot Heartbeat.sync_daily_brief/0; omitted when

| config :lemon_automation, daily_brief_sync: false)

+-- LemonAutomation.GoalContinuationManager (GenServer)

+-- LemonAutomation.GoalLoopManager (GenServer)

+-- LemonAutomation.KanbanDispatcher (GenServer)

+-- LemonAutomation.SkillCuratorManager (GenServer)

+-- LemonAutomation.SynthesisRunnerManager (GenServer)

```

There is no heartbeat process: heartbeats are cron jobs, and their suppression

runs synchronously inside `CronManager` terminalization.

## Cron System Architecture

```

CronManager (GenServer, ticks every 60s)

|

+- on tick: finds due jobs, claims scheduled run slots

|

+- execute_job/2

|

+- creates CronRun, persists or claims it in CronStore

+- emits :cron_run_started on "cron" bus

+- spawns Task via TaskSupervisor

|

+- RunCompletionWaiter.submit_and_wait/2

|

+- reads CronMemory, injects into prompt

+- pre-subscribes to Bus.run_topic(run_id)

+- forks session_key into sub-session

+- calls LemonRouter.submit(params)

+- calls RunCompletionWaiter.wait_already_subscribed/3

|

+- returns its result through the monitored Task ref

|

+- CronManager terminalizes: monitor/preflight policy, then

Heartbeat.apply_policy/2, updates CronStore, emits :cron_run_completed

+- If delivered: forwards a synthetic `:run_completed` summary into the

base main/channel-peer session topic/history, and sends the result to the

chat chosen by CronDelivery (deliver_to, originating chat, owner chat)

+- `once` jobs are deleted after their scheduled run (and retries) finish

```

**Wake** is a separate module (not intermediary in the above flow). It creates

runs with `triggered_by: :wake`, submits through the same configured

`:cron_run_submitter` used by `CronManager` (normally `RunSubmitter` ->

`LemonRouter`), and sends `{:run_complete, ...}` back to `CronManager`.

**CronCommandRunner** is the operator-owned no-agent cron path. Jobs with

`command` instead of `prompt` run as supervised local shell commands under

`CronManager`, persist output/error in `CronRun`, and never create LemonRouter

runs. They reach chat only with an explicit `deliver_to` target; otherwise

`CronDelivery.target/2` returns `:none` for them. Prompt jobs

still require `agent_id`, `session_key`, and `prompt`; command jobs require

only `name`, `schedule`, and `command`.

**GoalContinuationManager** runs one active persistent-goal continuation at a

time through `TaskSupervisor`. `GoalContinuation` submits normal

`LemonRouter` requests with `origin: :goal`, `queue_mode: :followup`, and goal

metadata, then records the returned run id in `LemonAgent.Workspace.GoalStore`.

**GoalLoopManager** runs goal-loop work through `TaskSupervisor`. `run_once/2`

performs one preview judge tick. `start_loop/2` starts one bounded autonomous

loop per session, waits for each submitted continuation to emit `:run_completed`,

and stops at `max_ticks`, failure, timeout, or a terminal judge verdict. Passing

`auto: true` persists opt-in auto scheduling in `LemonAgent.Workspace.GoalStore`; the

manager scheduler scans active goals and re-starts only persisted auto loops

when no loop for that session is already running. Focused tests cover that

persisted-auto path through the real goal loop and router judge runner.

Before each judge or continuation submission, the manager synchronously claims

the fixed router run id from `RunCompletionWaiter`. `stop_loop/2` defaults to

hard stop: it disables auto restart, aborts that owned router run once, kills

the loop task, and prevents another tick. The router serializes an abort

tombstone with submission acceptance, so a stop cannot miss a run between

acceptance and the later submitted callback. `mode: :graceful` disables auto

restart but lets the already bounded loop finish. The outer `run_once/2` call

timeout encloses the configured judge/continuation wait deadlines.

Before crossing the external abort boundary, hard stop durably records the

fixed run's abort intent and marks the attempt outcome unknown. A manager crash

after dispatch therefore restores reconciliation ownership and never repeats

the abort; a normal acknowledgement updates that prewritten intent.

The hard-stop result reports a sanitized `router_abort` status. It sends the

abort exactly once: `:outcome_unknown` means the abort may already have taken

effect, so the manager does not retry it; arbitrary callback errors and

exceptions never enter stored goal metadata or returned diagnostics.

An ambiguous judge or continuation submission remains owned by its

caller-generated run ID while the waiter reconciles terminal events. If that

bounded wait expires, crashes, or returns anything other than a confirmed

terminal result, the manager retains the loop in `reconciling` state

so another tick cannot overlap it. A durable terminal record releases that

guard into an error state requiring an explicit restart. A hard stop aborts

that exact run once; only a definite abort acceptance releases the guard, while

an ambiguous or unavailable abort keeps reconciliation ownership. Persisted

reconciliation and abort-attempt metadata restore that guard after a manager

restart without retrying the abort.

`GoalJudge` supports explicit verdicts, a pluggable `judge_runner` with

`judge_model` metadata, and deterministic fallback. `GoalJudge.RouterRunner`

is the dev/prod default runner; it submits isolated `:goal_judge` runs through

`LemonRouter`, waits for completion, and parses JSON verdicts. The focused

goal-loop tests prove this path through the real router, a router `RunProcess`,

and `RunCompletionWaiter` using a deterministic runtime. Set

`LEMON_GOAL_JUDGE_MODEL` to pin the model used for default judge runs. Judge

failures pause goals by default;

`judge_failure_policy: :continue_once` is the explicit fail-open path for one

continuation.

All automation paths that submit and then wait use

`RunCompletionWaiter.submit_and_wait/2`. It assigns a fixed run id and owns the

subscription from before router submission through terminal wait cleanup.

Goal loops also use its synchronous pre-submit ownership claim; a rejected

claim prevents the router call entirely.

Router run-id mismatches are explicit errors; callers never subscribe to a

replacement id after submission because a synchronous completion could already

have been lost. Cron, goal judge, autonomous goal continuation,

and Kanban worker paths share this lifecycle.

**KanbanDispatcher** is the first BEAM-native fleet-work supervisor. It scans

durable `LemonAgent.Workspace.KanbanStore` boards, reclaims expired leases, leases

dependency-unblocked tasks to a worker profile, runs a worker module through

`TaskSupervisor`, then records completion or failure back into the durable task.

Focused dispatcher tests cover bounded multi-worker leasing, completion,

explicit failure, crashed-worker failure marking, expired-lease reclaim, and a

production-shaped bounded-concurrency path through the real `KanbanRunWorker`

with router/waiter stubs. The default `KanbanRunWorker` submits leased tasks

through `LemonRouter` with `origin: :kanban`, board/task metadata, a blocked

`kanban` tool policy, and an isolated git worktree cwd when the board workspace

is a git repository. Worktrees are created under

`<repo>/.worktrees/kanban-<task_id>` on `lemon-kanban/<task_id>` branches, so

concurrent board workers do not edit the same checkout.

`stop_board/2` is a hard stop: owned worker pids are killed and their exact

lease IDs are reclaimed before return. Completion/failure writes are guarded by

lease ID, so late results cannot mutate a re-leased task. Starting a board also

reclaims unexpired leases left by the same dispatcher worker id after restart.

**Heartbeat** (`LemonAutomation.Heartbeat`) holds the heartbeat policy. A job

is a heartbeat when `meta.heartbeat == true` (atom or string key).

`CronManager.terminalize_run` calls `Heartbeat.apply_policy/2` after the monitor

and preflight policies: a `:completed` run whose output trims to exactly

`HEARTBEAT_OK` is marked suppressed (`CronRun.suppress/1`), emits

`:heartbeat_suppressed`, and is not delivered; any other outcome emits

`:heartbeat_alert` and is delivered. It also reconciles the daily brief:

`sync_daily_brief/0` reads `[automation] daily_brief` via

`LemonCore.Config.Modular.load().automation` and creates, reschedules, or

removes the job `cron_daily_brief` (`timezone: "local"`, agent `default`,

session `agent:default:main`, `meta: %{heartbeat: true, daily_brief: true}`).

Invalid times are logged and leave jobs untouched.

**CronDelivery** (`LemonAutomation.CronDelivery`) resolves the delivery target:

`deliver_to: "none"` → nothing; explicit `"telegram:<chat>[:<thread>]"` /

`"discord:<channel>[:<thread>]"`; the originating chat for channel-peer session

keys of registered chat platforms (sub-session suffix dropped, topic/thread

kept); otherwise the paired owner chat (first `gateway.telegram.allowed_chat_ids`

when Telegram is enabled, else `gateway.discord.default_channel_id` when Discord

is enabled); otherwise nothing. `validate/1` normalizes stored values

(`nil`/`""`/`"auto"` → default). Channel text is `"<job name>\n\n<output>"` or

`"<job name> (<status>): <error>"`, enqueued through

`LemonRouter.ChannelsDelivery` with idempotency key `cron_notify_<run_id>`.

**SkillCuratorManager** is the idle-triggered background path for learned-skill

maintenance. It checks active router sessions every minute, waits for the

configured idle window, then calls `LemonAutomation.SkillCurator`. That module

delegates lifecycle transitions and prompt rendering to `LemonSkills.Curator`

and submits the review prompt to `LemonRouter` only when review is required.

**Blueprint** is the portable distribution boundary for a coherent skill plus

automation workflow. It reuses `LemonSkills.Bundle`, manifest/lint/audit, the

derived `LemonCore.ProfileStore` workspace, and `CronManager`; do not add a

second skill registry, profile store, or scheduler. A preview digest binds the

manifest, content hashes, target profile, exact job projection, and current

collision state. Activation replans under a lock, stages and re-audits exact

copied bytes, enables profile-local skills, and calls `CronManager.add_new/1`

last. The create-once manager/store path must never overwrite an existing

stable ID. Version 1 remains agent-prompt-only and rejects commands, archives,

symlinks, environment/cwd overrides, secret-like values, and non-UTC schedules.

`LemonAutomation.Blueprint.Catalog` is the shared caller boundary for Web and

control-plane clients: it derives the canonical catalog path from trusted

profile options, accepts only bounded bundle IDs, rejects traversal and

symlinked entries, enforces manifest/directory identity, and delegates all

inspection, validation, preview, and activation semantics to `Blueprint`.

## Key Flow Details

- `RunSubmitter` pre-subscribes to `Bus.run_topic(run_id)` BEFORE submitting to `LemonRouter` (avoids race condition)

- `RunSubmitter` passes `run_id` in params so the router uses the same ID it already subscribed to

- `RunSubmitter` executes each cron run in a forked `:sub:<id>` session for isolation; the originating base session is preserved in run metadata

- `RunSubmitter` adds a cron-run tool policy with `blocked_tools: ["cron"]`; scheduled runs may do the task work, but they cannot recursively manage cron jobs through a cron tool

- `RunSubmitter` reads `CronMemory` for the job and injects memory context into the prompt, then appends run results back to the memory file

- If the router returns a different `run_id`, submission fails explicitly; no late replacement-id subscription is attempted

- Output is truncated to 1000 chars before storage

- Jobs execute in monitored `TaskSupervisor.async_nolink/2` tasks owned by `CronManager`; task-start failure and `:DOWN` terminalize the run immediately. Unsupervised fallback exists only when the manager is started with explicit `standalone_mode: true`.

- For cron jobs created from `agent:*:main`, completion summaries are mirrored back into the base main session as synthetic `run_completed` entries (`meta.cron_forwarded_summary = true`)

- Scheduled ticks and `CronManager` restarts consult persisted active runs before launching; active runs older than the job timeout recover through the same monitor/delivery/retry terminalizer as normal completions.

- Scheduled runs use deterministic slot ids and `CronStore.claim_scheduled_run/3`, backed by `LemonCore.Store.put_new/3`, so competing dispatchers preserve the first claimant instead of overwriting a run.

- Scheduled failures/timeouts persist retry due time, attempt, source, and root lineage on the terminal source run. Restart reconstructs the timer, while a deterministic `{root, attempt}` retry-run ID and atomic claim prevent duplicate attempts. Manual and wake runs stay single-shot by default.

- Active cron runs can be aborted by cron run id. `CronManager.abort_run/1` calls the underlying router cancellation when possible, persists terminal `:aborted`, emits the normal completion event, and ignores late submitter completions.

- Cron lifecycle actions write durable operator audit events to `:cron_audit_events`. The audit stream covers job create/update/pause/resume/delete, manual run requests, run start/abort/retry/stale recovery, and scheduled-run claim/suppression decisions. Audit entries keep operator-useful IDs in the store; support-bundle diagnostics redact those IDs.

- `cron.status` reads the durable cron run and audit stores directly for operator-facing scheduler-health counters: active run locks, retry runs, suppressed scheduled slots, stale-run recoveries, scheduled retries, and next/last run timestamps.

- Portable bundle activation must remain preview-first and content-free at the

Web/control-plane boundary. Keep arbitrary paths local-only;

`Blueprint.Catalog` resolves a safe `bundleId` below `~/.lemon/bundles`,

returns no prompt/skill/path/secret text,

and requires the exact fresh `confirmationDigest` for mutation.

- Source and packaged `lemon blueprints` are thin clients of that RPC boundary.

Do not move bundle loading or scheduler startup into the one-shot CLI VM;

preview remains the bundle-ID shorthand and activation remains an explicit

exact-digest request handled by the long-running `CronManager` runtime.

## Top-Level Facade

`LemonAutomation` module provides delegating functions:

```elixir

LemonAutomation.list_jobs() # delegates to CronManager.list/0

LemonAutomation.add_job(params) # delegates to CronManager.add/1

LemonAutomation.update_job(id, params) # delegates to CronManager.update/2

LemonAutomation.remove_job(id) # delegates to CronManager.remove/1

LemonAutomation.run_now(id) # delegates to CronManager.run_now/1

LemonAutomation.abort_run(run_id) # delegates to CronManager.abort_run/1

LemonAutomation.runs(job_id, opts) # delegates to CronManager.runs/2

LemonAutomation.wake(job_id) # delegates to Wake.trigger/1

```

## How to Add a Cron Job

### Basic Job

```elixir

{:ok, job} = LemonAutomation.CronManager.add(%{

name: "Daily Report",

schedule: "0 9 * * *", # 9 AM daily

agent_id: "agent_abc",

session_key: "agent:agent_abc:main",

prompt: "Generate daily status report"

})

```

### Command Job

```elixir

{:ok, job} = LemonAutomation.CronManager.add(%{

name: "Disk Usage Snapshot",

schedule: "hourly",

command: "df -h /",

cwd: "/tmp"

})

```

### With Timezone and Jitter

```elixir

{:ok, job} = LemonAutomation.CronManager.add(%{

name: "Hourly Check",

schedule: "0 * * * *",

agent_id: "agent_abc",

session_key: "agent:agent_abc:main",

prompt: "Check system status",

timezone: "America/New_York", # Default: "UTC"

jitter_sec: 30, # Random 0-30s delay before execution

max_retries: 2, # Scheduled failure/timeout retries

retry_backoff_ms: 60_000 # Delay before each retry

})

```

### With Persistent Memory

```elixir

{:ok, job} = LemonAutomation.CronManager.add(%{

name: "Weekly Summary",

schedule: "0 9 * * 1",

agent_id: "agent_abc",

session_key: "agent:agent_abc:main",

prompt: "Generate weekly summary using notes from previous runs",

memory_file: "/path/to/custom/memory.md" # Optional; auto-generated if omitted

})

```

### Required Fields

| Field | Description |

|-------|-------------|

| `name` | Human-readable identifier |

| `schedule` | Cron expression (5 fields) or supported shorthand |

| `agent_id` | Target agent for prompt jobs |

| `session_key` | Routing key for prompt jobs (e.g., `"agent:{id}:main"`) |

| `prompt` | Text sent to agent; mutually exclusive with `command` |

| `command` | Local shell command for no-agent operator jobs |

Updates preserve target type. Prompt jobs can update `prompt`; command jobs can

update `command`, `cwd`, and `env`; `agent_id` and `session_key` stay immutable.

### Optional Fields

| Field | Default | Description |

|-------|---------|-------------|

| `enabled` | `true` | Whether the job is active |

| `timezone` | `"UTC"` | Timezone for schedule interpretation |

| `jitter_sec` | `0` | Random delay spread in seconds |

| `timeout_ms` | `300_000` | Max execution time (5 minutes) |

| `max_retries` | `0` | Number of retries after scheduled failure/timeout |

| `retry_backoff_ms` | `30_000` | Delay before retry runs |

| `cwd` | runtime cwd | Command working directory |

| `env` | `%{}` | Command environment overrides |

| `memory_file` | auto | Path to persistent cross-run memory file |

| `meta` | `nil` | Arbitrary metadata map |

### Cron Expression Format

```

* * * * *

| | | | +-- Day of week (0-7, 0/7 = Sunday)

| | | +---- Month (1-12)

| | +------ Day of month (1-31)

| +-------- Hour (0-23)

+---------- Minute (0-59)

```

Supported cron syntax: `*`, `N`, `N-M` (range), `*/N` (step), `N,M,O` (list).

Supported shorthands are normalized into 5-field cron expressions before

storage. Interval shorthands must map cleanly to cron step fields, so minute

intervals divide 60 and hour intervals divide 24:

- `"every 30m"`, `"15 minutes"` -> `"*/30 * * * *"`, `"*/15 * * * *"`

- `"hourly"`, `"every 2h"` -> `"0 * * * *"`, `"0 */2 * * *"`

- `"daily at 9am"`, `"every day at 17:45"` -> `"0 9 * * *"`, `"45 17 * * *"`

- `"weekdays at 09:30"` -> `"30 9 * * 1-5"`

- `"weekly monday at 8am"`, `"fridays at 18:15"` -> `"0 8 * * 1"`, `"15 18 * * 5"`

Examples:

- `"*/15 * * * *"` - Every 15 minutes

- `"0 9 * * 1"` - Mondays at 9 AM

- `"0 */6 * * *"` - Every 6 hours

### Managing Jobs

```elixir

# List all jobs (sorted by created_at_ms desc)

LemonAutomation.CronManager.list()

# Update a job (agent_id and session_key are immutable; attempts return {:error, {:immutable_fields, [...]}})

LemonAutomation.CronManager.update(job.id, %{enabled: false})

# Remove a job (also clears heartbeat config if job has meta.heartbeat: true)

LemonAutomation.CronManager.remove(job.id)

# Run immediately (manual trigger, job must exist but disabled jobs still work)

LemonAutomation.CronManager.run_now(job.id)

# Get run history

# opts: limit (default 100), status (atom filter), since_ms (timestamp filter)

LemonAutomation.CronManager.runs(job.id, limit: 10)

# Force a tick cycle (for testing)

LemonAutomation.CronManager.tick() # cast, returns :ok immediately

```

## CronJob Struct

```elixir

%LemonAutomation.CronJob{

id: "cron_abc123", # auto-generated via LemonCore.Id.cron_id()

name: "Daily Report",

schedule: "0 9 * * *",

enabled: true,

agent_id: "agent_abc",

session_key: "agent:agent_abc:main",

prompt: "Generate daily status report",

memory_file: nil, # auto-generated path if nil

timezone: "UTC",

jitter_sec: 0,

timeout_ms: 300_000,

max_retries: 0,

retry_backoff_ms: 30_000,

created_at_ms: 1739989200000,

updated_at_ms: 1739989200000,

last_run_at_ms: nil,

next_run_at_ms: 1739989200000,

meta: nil

}

```

Key functions: `CronJob.new/1`, `CronJob.update/2`, `CronJob.due?/1`, `CronJob.mark_run/2`, `CronJob.set_next_run/2`, `CronJob.to_map/1`, `CronJob.from_map/1`.

## CronRun Struct and State Machine

```elixir

%LemonAutomation.CronRun{

id: "run_abc123", # auto-generated via LemonCore.Id.run_id()

job_id: "cron_xyz789",

run_id: nil, # LemonRouter run ID (set after submission)

status: :pending, # :pending | :running | :completed | :failed | :timeout | :aborted

triggered_by: :schedule, # :schedule | :manual | :wake | :retry

started_at_ms: nil,

completed_at_ms: nil,

duration_ms: nil,

output: nil, # truncated to 1000 chars

error: nil,

suppressed: false, # true if heartbeat with "HEARTBEAT_OK" response

meta: nil

}

```

State transitions:

```

CronRun.new(job_id, triggered_by) => status: :pending

CronRun.start(run, run_id \\ nil) => status: :running

CronRun.complete(run, output) => status: :completed

CronRun.fail(run, error) => status: :failed

CronRun.timeout(run) => status: :timeout

CronRun.abort(run, reason) => status: :aborted

CronRun.suppress(run) => suppressed: true (can combine with any terminal state)

```

Helper predicates: `CronRun.active?/1` (pending or running), `CronRun.finished?/1`.

Scheduled failures and timeouts retry only when `max_retries > 0`. The terminal

source run persists `retry_due_at_ms`, `retry_next_attempt`, `retry_of`, and

`retry_root_id`; each claimed retry is a separate run with

`triggered_by: :retry`. Deterministic retry IDs make one root/attempt pair a

stable uniqueness slot across manager restarts. Manual and wake-triggered runs

do not retry by default.

## CronMemory (Persistent Cross-Run Memory)

Each cron job can have a persistent markdown memory file that accumulates context across runs:

- Default location: `~/.lemon/cron_memory/{job_id}.md`

- Configurable via `memory_file` field on the job or `meta.memory_file`/`meta.memoryFile`

- On each run, `RunSubmitter` reads the memory file and injects it into the prompt

- After each run, the result is appended to the memory file

- Auto-compaction kicks in at 24,000 chars: older content is summarized, recent 14,000 chars retained

- Prompt injection limited to 8,000 chars (tail of file)

## Heartbeat Management

Heartbeats are ordinary cron jobs; there is no heartbeat process, timer path,

or separate store. `LemonAutomation.Heartbeat` supplies the policy that

`CronManager` applies synchronously during run terminalization. Heartbeat

schedules are cron schedules, so intervals must be cron-exact.

### Identifying Heartbeat Jobs

A job is a heartbeat only when its `meta` has `heartbeat: true` (atom or string

key). The job name is not consulted.

### Creating a Heartbeat via CronManager

```elixir

{:ok, job} = LemonAutomation.CronManager.add(%{

name: "heartbeat-agent_abc",

schedule: "every 30m",

timezone: "local",

agent_id: "agent_abc",

session_key: "agent:agent_abc:main",

prompt: "Work through HEARTBEAT.md; reply HEARTBEAT_OK if nothing needs attention.",

meta: %{heartbeat: true, agent_id: "agent_abc"}

})

```

The TUI `/heartbeat every <interval> <prompt>` command creates or updates this

per-agent job (it ignores `meta.daily_brief` jobs).

### Heartbeat API

```elixir

# meta.heartbeat == true (atom or string key)

LemonAutomation.Heartbeat.heartbeat?(job)

# Exact "HEARTBEAT_OK" after trimming

LemonAutomation.Heartbeat.healthy_response?(response)

# Called by CronManager terminalization; returns {run, deliver?}

LemonAutomation.Heartbeat.apply_policy(job, run)

# Daily brief: reconcile cron_daily_brief from [automation] daily_brief

LemonAutomation.Heartbeat.sync_daily_brief()

# ...or from an explicit HH:MM (nil removes the job); opts[:cron_manager] for tests

LemonAutomation.Heartbeat.sync_daily_brief("08:00", cron_manager: MyCron)

```

### Heartbeat Suppression Behavior

- Runs after the monitor and preflight policies in `CronManager.terminalize_run`

- `:completed` + exact `HEARTBEAT_OK`: `CronRun.suppress/1`, kept in run history, emits `:heartbeat_suppressed`, NOT delivered

- Any other outcome (including failure/timeout): emits `:heartbeat_alert` with `severity: :warning`, delivered through `CronDelivery`

- Non-heartbeat jobs pass through unchanged

### Daily Brief

`[automation] daily_brief = "HH:MM"` (`LemonCore.Config.Automation`) is

reconciled once at boot by the `:daily_brief_sync` task into job

`cron_daily_brief`: schedule `"M H * * *"`, `timezone: "local"`, agent

`default`, session `agent:default:main`, `meta: %{heartbeat: true,

daily_brief: true}`. A changed time updates schedule/timezone and re-enables

the job; removing the key removes it; an invalid time is logged and returns

`{:error, :invalid_time}` without touching jobs. `config :lemon_automation,

daily_brief_sync: false` (used in `config/test.exs`) omits the task.

## Wake Functionality

Wake triggers immediate job execution outside the normal schedule.

```elixir

# Trigger single job (must be enabled, returns {:error, :job_disabled} otherwise)

{:ok, run} = LemonAutomation.Wake.trigger("cron_abc123")

# Trigger with context (stored in run.meta.wake_context)

{:ok, run} = LemonAutomation.Wake.trigger("cron_abc123",

context: %{reason: "incident response"}

)

# Skip if already running (checks CronStore.active_runs/1)

{:ok, run} = LemonAutomation.Wake.trigger("cron_abc123",

skip_if_running: true

)

# Returns {:error, :already_running} if active runs exist

# Trigger multiple jobs (returns map of job_id => result)

results = LemonAutomation.Wake.trigger_many(["cron_abc", "cron_xyz"])

# Trigger by name pattern (case-insensitive substring match on enabled jobs)

results = LemonAutomation.Wake.trigger_matching("heartbeat")

# Trigger all enabled jobs for an agent

results = LemonAutomation.Wake.trigger_for_agent("agent_abc")

```

Wake runs use `triggered_by: :wake` and fire-and-forget: they return the

`CronRun` immediately, use the same configured `:cron_run_submitter` as

`CronManager`, and have completion handled asynchronously by `CronManager`.

Wake failures do not enter the scheduled retry path.

## Events

All events are broadcast on the `"cron"` bus topic as `%LemonCore.Event{}` structs.

### Event Types

| Event | Emitted By | When |

|-------|-----------|------|

| `:cron_tick` | `CronManager` | Every 60s tick |

| `:cron_job_created` | `CronManager` | Job added |

| `:cron_job_updated` | `CronManager` | Job updated |

| `:cron_job_deleted` | `CronManager` | Job removed |

| `:cron_run_started` | `CronManager`, `Wake` | Run begins |

| `:cron_run_completed` | `CronManager` | Run finishes (terminalization) |

| `:cron_lifecycle_action` | `CronManager` | Durable audit event recorded |

| `:heartbeat_suppressed` | `Heartbeat.apply_policy/2` (CronManager terminalization) | "HEARTBEAT_OK" response |

| `:heartbeat_alert` | `Heartbeat.apply_policy/2` (CronManager terminalization) | Any other heartbeat outcome |

### Subscribing to Events

```elixir

LemonCore.Bus.subscribe("cron")

receive do

%LemonCore.Event{type: :cron_run_started, payload: payload} ->

IO.puts("Run started: #{payload.run.id}")

%LemonCore.Event{type: :cron_run_completed, payload: payload} ->

IO.puts("Run completed: #{payload.run.status}, suppressed: #{payload.suppressed}")

%LemonCore.Event{type: :cron_lifecycle_action, payload: payload} ->

IO.puts("Audit action: #{payload.audit.action}")

%LemonCore.Event{type: :heartbeat_suppressed, payload: payload} ->

IO.puts("Heartbeat OK for job #{payload.job_id}")

%LemonCore.Event{type: :heartbeat_alert, payload: payload} ->

IO.puts("Heartbeat alert for agent #{payload.agent_id}: #{payload.response}")

end

```

## CronStore API

```elixir

# Jobs (tables: :cron_jobs)

CronStore.put_job(job)

CronStore.get_job(job_id)

CronStore.delete_job(job_id)

CronStore.list_jobs() # all jobs, sorted by created_at_ms desc

CronStore.list_enabled_jobs() # only enabled: true

CronStore.list_due_jobs() # enabled and due?(job) == true

# Runs (table: :cron_runs)

CronStore.put_run(run)

CronStore.claim_run(run)

CronStore.claim_scheduled_run(job, scheduled_for_ms, router_run_id)

CronStore.claim_retry_run(job, source_run, router_run_id)

CronStore.retry_run_id(root_run_id, attempt)

CronStore.pending_retries()

CronStore.get_run(run_id)

CronStore.delete_run(run_id)

CronStore.list_runs(job_id, opts) # opts: limit (100), status (atom), since_ms

CronStore.list_all_runs(opts) # across all jobs

CronStore.active_runs(job_id) # runs where status in [:pending, :running]

CronStore.cleanup_old_runs(keep_per_job \\ 100)

# Audit (table: :cron_audit_events)

CronStore.record_audit(action, attrs)

CronStore.list_audit_events(opts) # opts: limit, job_id, run_id, action, since_ms

CronStore.delete_audit_event(id)

```

## CronSchedule API

```elixir

# Parse expression into structured map

{:ok, parsed} = CronSchedule.parse("*/15 * * * *")

# parsed = %{minute: [0,15,30,45], hour: [0..23], day: [1..31], month: [1..12], weekday: [0..6]}

# Get next run timestamp in milliseconds

CronSchedule.next_run_ms("0 9 * * *", "UTC") # => integer ms | nil

CronSchedule.next_run_datetime("0 9 * * *", "UTC") # => %DateTime{} | nil

# Get multiple future run times

CronSchedule.next_runs("*/15 * * * *", "UTC", count: 5) # => [%DateTime{}, ...]

# Check if a DateTime matches an expression

CronSchedule.matches?("0 9 * * *", datetime) # => boolean

# Validate expression

CronSchedule.valid?("0 9 * * *") # => boolean

```

## Common Tasks

### Check Job Status

```elixir

alias LemonAutomation.{CronStore, CronJob}

# Check if job is currently due

job = CronStore.get_job("cron_abc123")

CronJob.due?(job)

# Get active runs

active = CronStore.active_runs("cron_abc123")

# Get last run

[last | _] = CronStore.list_runs("cron_abc123", limit: 1)

```

### Clean Up Old Runs

```elixir

# Keep only last 50 runs per job (default: 100)

LemonAutomation.CronStore.cleanup_old_runs(50)

```

## Testing Guidance

### Test Structure

```

test/lemon_automation/

+-- cron_job_test.exs # CronJob struct lifecycle

+-- cron_run_test.exs # CronRun state transitions

+-- cron_schedule_test.exs # Cron parsing, next_run computation

+-- cron_store_test.exs # Persistence operations

+-- cron_manager_update_test.exs # Immutable field rejection

+-- cron_manager_forwarding_test.exs # Base-session forwarding, owner-chat default, deliver_to, heartbeat suppression, one-shot removal

+-- cron_delivery_test.exs # deliver_to validation, target resolution, channel text

+-- events_test.exs # Event emission

+-- heartbeat_test.exs # apply_policy/2 and daily-brief reconciliation

+-- run_completion_waiter_test.exs # Wait logic

+-- run_submitter_test.exs # Router submission

+-- goal_loop_test.exs # Goal loops and router judge proof

+-- goal_judge_router_live_test.exs # Opt-in provider-backed judge proof

+-- kanban_dispatcher_test.exs # Dispatcher leasing/concurrency/failure proof

+-- kanban_dispatcher_live_test.exs # Opt-in provider-backed dispatcher proof

+-- kanban_run_worker_test.exs # Router-backed kanban worker proof

+-- wake_test.exs # Wake triggering

```

### Running Tests

```bash

# All automation tests

mix test apps/lemon_automation

# Specific module

mix test apps/lemon_automation/test/lemon_automation/cron_schedule_test.exs

# Provider-backed goal-judge proof

ZAI_API_KEY="$(MIX_ENV=dev mix run --no-start -e 'Logger.configure(level: :emergency); Logger.remove_backend(:console); {:ok, _} = Application.ensure_all_started(:lemon_core); IO.write(LemonCore.Secrets.fetch_value("llm_zai_api_key") || "")')" \

LEMON_TEST_ALLOW_LIVE_CREDENTIALS=1 \

LEMON_GOAL_JUDGE_MODEL="zai:glm-5-turbo" \

scripts/test path apps/lemon_automation/test/lemon_automation/goal_judge_router_live_test.exs --include integration --seed 1

# Provider-backed kanban dispatcher proof

secret_file=$(mktemp "${TMPDIR:-/tmp}/lemon-zai-secret.XXXXXX")

LEMON_SECRET_OUTPUT="$secret_file" MIX_ENV=dev mix run --no-start -e '

Logger.configure(level: :emergency)

Logger.remove_backend(:console)

{:ok, _} = Application.ensure_all_started(:lemon_core)

case LemonCore.Secrets.fetch_value("llm_zai_api_key") do

value when is_binary(value) and value != "" -> File.write!(System.fetch_env!("LEMON_SECRET_OUTPUT"), value)

_ -> System.halt(66)

end

' >/dev/null 2>&1

ZAI_API_KEY="$(cat "$secret_file")" \

LEMON_TEST_ALLOW_LIVE_CREDENTIALS=1 \

LEMON_KANBAN_LIVE_MODEL="zai:glm-5-turbo" \

scripts/test path apps/lemon_automation/test/lemon_automation/kanban_dispatcher_live_test.exs --include integration --seed 1

rc=$?

rm -f "$secret_file"

exit "$rc"

# With coverage

mix test --cover apps/lemon_automation

# Portable bundle policy, rollback, exact confirmation, and replay

mix test apps/lemon_automation/test/lemon_automation/blueprint_test.exs --seed 1

```

### Key Testing Patterns

**Mock the Router:**

```elixir

test "submits to router", %{job: job, run: run} do

defmodule MockRouter do

def submit(params) do

send(self(), {:submitted, params})

{:ok, "run_123"}

end

end

result = RunSubmitter.submit(job, run,

router_mod: MockRouter,

waiter_mod: MockWaiter

)

end

```

**Test Cron Parsing:**

```elixir

test "parses valid cron expressions" do

assert {:ok, parsed} = CronSchedule.parse("*/15 * * * *")

assert parsed.minute == [0, 15, 30, 45]

end

```

**Test Heartbeat Suppression:**

```elixir

test "suppresses exact HEARTBEAT_OK" do

# Exact match and whitespace-trimmed variants are suppressed

assert Heartbeat.healthy_response?("HEARTBEAT_OK")

assert Heartbeat.healthy_response?(" HEARTBEAT_OK ")

assert Heartbeat.healthy_response?("HEARTBEAT_OK\n")

# Any other string is NOT suppressed

refute Heartbeat.healthy_response?("Status: OK")

refute Heartbeat.healthy_response?("HEARTBEAT_OK extra")

refute Heartbeat.healthy_response?(nil)

end

```

**Test State Transitions:**

```elixir

test "run state transitions" do

run = CronRun.new("job_1", :schedule)

assert run.status == :pending

run = CronRun.start(run)

assert run.status == :running

run = CronRun.complete(run, "output")

assert run.status == :completed

run = CronRun.suppress(run)

assert run.suppressed == true

end

```

**Force a Tick:**

```elixir

# Cast-based tick (returns immediately, execution is async)

LemonAutomation.CronManager.tick()

```

## Dependencies

- `lemon_core` - Store, Bus, Clock, Id generation

- `lemon_router` - Run submission via `LemonRouter.submit/1`

- `jason` - JSON serialization

## Key Modules Reference

| Module | Purpose |

|--------|---------|

| `LemonAutomation` | Top-level facade with delegating functions |

| `LemonAutomation.Application` | OTP supervisor (TaskSupervisor, CronManager, daily-brief sync task, goal/kanban/curator/synthesis managers) |

| `LemonAutomation.Blueprint` | Portable manifest validation, exact preview confirmation, profile-local skill activation, and idempotent cron claim |

| `LemonAutomation.CronManager` | Scheduling GenServer; owns job state in-memory + persists to CronStore |

| `LemonAutomation.CronJob` | Job struct, CRUD ops, `due?/1` predicate |

| `LemonAutomation.CronRun` | Run struct, state machine transitions |

| `LemonAutomation.CronSchedule` | Cron parsing, next-run computation, `valid?/1`, `matches?/2` |

| `LemonAutomation.CronStore` | Persistence via LemonCore.Store (tables: `:cron_jobs`, `:cron_runs`) |

| `LemonAutomation.CronMemory` | Persistent markdown-based cross-run memory; auto-compaction |

| `LemonAutomation.Heartbeat` | Heartbeat policy (`meta.heartbeat`, `HEARTBEAT_OK` suppression) applied in CronManager terminalization; daily-brief reconciliation |

| `LemonAutomation.CronDelivery` | Delivery target resolution (`deliver_to`, originating chat, paired owner chat), channel text, outbox enqueue |

| `LemonAutomation.SkillCurator` | Idle/config gates plus background submission for learned-skill curator prompts with a learning-only default tool policy |

| `LemonAutomation.SkillCuratorManager` | Periodic idle scheduler for `SkillCurator.run_once/1` |

| `LemonAutomation.Wake` | Manual immediate triggering (fire-and-forget, enabled jobs only) |

| `LemonAutomation.RunCompletionWaiter` | Owns race-free fixed-id submit/wait subscriptions and parses terminal events |

| `LemonAutomation.RunSubmitter` | Builds params, pre-subscribes to bus, submits to LemonRouter, manages CronMemory |

| `LemonAutomation.Events` | Event emission helpers for all automation events |