git:20260927.979dcab to git:20260927.6027bc7

- # LemonAutomation

-

- Elixir app for cron jobs, heartbeats, and automation tasks in the Lemon umbrella.

-

- ## Purpose and Responsibilities

-

- LemonAutomation provides scheduled and triggered automation for agents:

-

- - **Cron Jobs** - Schedule agent prompts with cron expressions

- - **Heartbeats** - Cron jobs flagged `meta.heartbeat`; exact `HEARTBEAT_OK` replies are suppressed, everything else is delivered

- - **Delivery** - Finished run results go to an explicit `deliver_to` target, the originating chat, or the paired owner chat (`CronDelivery`)

- - **Daily Brief** - Opt-in `[automation] daily_brief` heartbeat reconciled at boot

- - **Wake** - Immediate manual triggering of scheduled jobs

- - **Goal Continuation** - Supervised one-shot continuation runs for active session goals

- - **Goal Loop Preview** - One supervised judge tick that continues, completes, or pauses a goal

- - **Kanban Dispatch** - Supervised leasing/reclaim loop for durable board tasks

- - **Portable Blueprints** - Preview and explicitly confirm profile-local skill bundles plus one agent cron job

- - **Run Tracking** - Full lifecycle tracking of job executions

-

- ## Supervision Tree

-

- ```

- LemonAutomation.Supervisor (one_for_one)

- +-- LemonAutomation.TaskSupervisor (Task.Supervisor)

- +-- LemonAutomation.CronManager (GenServer; runs Heartbeat policy + CronDelivery at terminalization)

- +-- Task :daily_brief_sync (one-shot Heartbeat.sync_daily_brief/0; omitted when

- | config :lemon_automation, daily_brief_sync: false)

- +-- LemonAutomation.GoalContinuationManager (GenServer)

- +-- LemonAutomation.GoalLoopManager (GenServer)

- +-- LemonAutomation.KanbanDispatcher (GenServer)

- +-- LemonAutomation.SkillCuratorManager (GenServer)

- +-- LemonAutomation.SynthesisRunnerManager (GenServer)

- ```

-

- There is no heartbeat process: heartbeats are cron jobs, and their suppression

- runs synchronously inside `CronManager` terminalization.

-

- ## Cron System Architecture

-

- ```

- CronManager (GenServer, ticks every 60s)

- |

- +- on tick: finds due jobs, claims scheduled run slots

- |

- +- execute_job/2

- |

- +- creates CronRun, persists or claims it in CronStore

- +- emits :cron_run_started on "cron" bus

- +- spawns Task via TaskSupervisor

- |

- +- RunCompletionWaiter.submit_and_wait/2

- |

- +- reads CronMemory, injects into prompt

- +- pre-subscribes to Bus.run_topic(run_id)

- +- forks session_key into sub-session

- +- calls LemonRouter.submit(params)

- +- calls RunCompletionWaiter.wait_already_subscribed/3

- |

- +- returns its result through the monitored Task ref

- |

- +- CronManager terminalizes: monitor/preflight policy, then

- Heartbeat.apply_policy/2, updates CronStore, emits :cron_run_completed

- +- If delivered: forwards a synthetic `:run_completed` summary into the

- base main/channel-peer session topic/history, and sends the result to the

- chat chosen by CronDelivery (deliver_to, originating chat, owner chat)

- +- `once` jobs are deleted after their scheduled run (and retries) finish

- ```

-

- **Wake** is a separate module (not intermediary in the above flow). It creates

- runs with `triggered_by: :wake`, submits through the same configured

- `:cron_run_submitter` used by `CronManager` (normally `RunSubmitter` ->

- `LemonRouter`), and sends `{:run_complete, ...}` back to `CronManager`.

-

- **CronCommandRunner** is the operator-owned no-agent cron path. Jobs with

- `command` instead of `prompt` run as supervised local shell commands under

- `CronManager`, persist output/error in `CronRun`, and never create LemonRouter

- runs. They reach chat only with an explicit `deliver_to` target; otherwise

- `CronDelivery.target/2` returns `:none` for them. Prompt jobs

- still require `agent_id`, `session_key`, and `prompt`; command jobs require

- only `name`, `schedule`, and `command`.

-

- **GoalContinuationManager** runs one active persistent-goal continuation at a

- time through `TaskSupervisor`. `GoalContinuation` submits normal

- `LemonRouter` requests with `origin: :goal`, `queue_mode: :followup`, and goal

- metadata, then records the returned run id in `LemonAgent.Workspace.GoalStore`.

-

- **GoalLoopManager** runs goal-loop work through `TaskSupervisor`. `run_once/2`

- performs one preview judge tick. `start_loop/2` starts one bounded autonomous

- loop per session, waits for each submitted continuation to emit `:run_completed`,

- and stops at `max_ticks`, failure, timeout, or a terminal judge verdict. Passing

- `auto: true` persists opt-in auto scheduling in `LemonAgent.Workspace.GoalStore`; the

- manager scheduler scans active goals and re-starts only persisted auto loops

- when no loop for that session is already running. Focused tests cover that

- persisted-auto path through the real goal loop and router judge runner.

- Before each judge or continuation submission, the manager synchronously claims

- the fixed router run id from `RunCompletionWaiter`. `stop_loop/2` defaults to

- hard stop: it disables auto restart, aborts that owned router run once, kills

- the loop task, and prevents another tick. The router serializes an abort

- tombstone with submission acceptance, so a stop cannot miss a run between

- acceptance and the later submitted callback. `mode: :graceful` disables auto

- restart but lets the already bounded loop finish. The outer `run_once/2` call

- timeout encloses the configured judge/continuation wait deadlines.

-

- Before crossing the external abort boundary, hard stop durably records the

- fixed run's abort intent and marks the attempt outcome unknown. A manager crash

- after dispatch therefore restores reconciliation ownership and never repeats

- the abort; a normal acknowledgement updates that prewritten intent.

-

- The hard-stop result reports a sanitized `router_abort` status. It sends the

- abort exactly once: `:outcome_unknown` means the abort may already have taken

- effect, so the manager does not retry it; arbitrary callback errors and

- exceptions never enter stored goal metadata or returned diagnostics.

-

- An ambiguous judge or continuation submission remains owned by its

- caller-generated run ID while the waiter reconciles terminal events. If that

- bounded wait expires, crashes, or returns anything other than a confirmed

- terminal result, the manager retains the loop in `reconciling` state

- so another tick cannot overlap it. A durable terminal record releases that

- guard into an error state requiring an explicit restart. A hard stop aborts

- that exact run once; only a definite abort acceptance releases the guard, while

- an ambiguous or unavailable abort keeps reconciliation ownership. Persisted

- reconciliation and abort-attempt metadata restore that guard after a manager

- restart without retrying the abort.

-

- `GoalJudge` supports explicit verdicts, a pluggable `judge_runner` with

- `judge_model` metadata, and deterministic fallback. `GoalJudge.RouterRunner`

- is the dev/prod default runner; it submits isolated `:goal_judge` runs through

- `LemonRouter`, waits for completion, and parses JSON verdicts. The focused

- goal-loop tests prove this path through the real router, a router `RunProcess`,

- and `RunCompletionWaiter` using a deterministic runtime. Set

- `LEMON_GOAL_JUDGE_MODEL` to pin the model used for default judge runs. Judge

- failures pause goals by default;

- `judge_failure_policy: :continue_once` is the explicit fail-open path for one

- continuation.

-

- All automation paths that submit and then wait use

- `RunCompletionWaiter.submit_and_wait/2`. It assigns a fixed run id and owns the

- subscription from before router submission through terminal wait cleanup.

- Goal loops also use its synchronous pre-submit ownership claim; a rejected

- claim prevents the router call entirely.

- Router run-id mismatches are explicit errors; callers never subscribe to a

- replacement id after submission because a synchronous completion could already

- have been lost. Cron, goal judge, autonomous goal continuation,

- and Kanban worker paths share this lifecycle.

-

- **KanbanDispatcher** is the first BEAM-native fleet-work supervisor. It scans

- durable `LemonAgent.Workspace.KanbanStore` boards, reclaims expired leases, leases

- dependency-unblocked tasks to a worker profile, runs a worker module through

- `TaskSupervisor`, then records completion or failure back into the durable task.

- Focused dispatcher tests cover bounded multi-worker leasing, completion,

- explicit failure, crashed-worker failure marking, expired-lease reclaim, and a

- production-shaped bounded-concurrency path through the real `KanbanRunWorker`

- with router/waiter stubs. The default `KanbanRunWorker` submits leased tasks

- through `LemonRouter` with `origin: :kanban`, board/task metadata, a blocked

- `kanban` tool policy, and an isolated git worktree cwd when the board workspace

- is a git repository. Worktrees are created under

- `<repo>/.worktrees/kanban-<task_id>` on `lemon-kanban/<task_id>` branches, so

- concurrent board workers do not edit the same checkout.

- `stop_board/2` is a hard stop: owned worker pids are killed and their exact

- lease IDs are reclaimed before return. Completion/failure writes are guarded by

- lease ID, so late results cannot mutate a re-leased task. Starting a board also

- reclaims unexpired leases left by the same dispatcher worker id after restart.

-

- **Heartbeat** (`LemonAutomation.Heartbeat`) holds the heartbeat policy. A job

- is a heartbeat when `meta.heartbeat == true` (atom or string key).

- `CronManager.terminalize_run` calls `Heartbeat.apply_policy/2` after the monitor

- and preflight policies: a `:completed` run whose output trims to exactly

- `HEARTBEAT_OK` is marked suppressed (`CronRun.suppress/1`), emits

- `:heartbeat_suppressed`, and is not delivered; any other outcome emits

- `:heartbeat_alert` and is delivered. It also reconciles the daily brief:

- `sync_daily_brief/0` reads `[automation] daily_brief` via

- `LemonCore.Config.Modular.load().automation` and creates, reschedules, or

- removes the job `cron_daily_brief` (`timezone: "local"`, agent `default`,

- session `agent:default:main`, `meta: %{heartbeat: true, daily_brief: true}`).

- Invalid times are logged and leave jobs untouched.

-

- **CronDelivery** (`LemonAutomation.CronDelivery`) resolves the delivery target:

- `deliver_to: "none"` → nothing; explicit `"telegram:<chat>[:<thread>]"` /

- `"discord:<channel>[:<thread>]"`; the originating chat for channel-peer session

- keys of registered chat platforms (sub-session suffix dropped, topic/thread

- kept); otherwise the paired owner chat (first `gateway.telegram.allowed_chat_ids`

- when Telegram is enabled, else `gateway.discord.default_channel_id` when Discord

- is enabled); otherwise nothing. `validate/1` normalizes stored values

- (`nil`/`""`/`"auto"` → default). Channel text is `"<job name>\n\n<output>"` or

- `"<job name> (<status>): <error>"`, enqueued through

- `LemonRouter.ChannelsDelivery` with idempotency key `cron_notify_<run_id>`.

-

- **SkillCuratorManager** is the idle-triggered background path for learned-skill

- maintenance. It checks active router sessions every minute, waits for the

- configured idle window, then calls `LemonAutomation.SkillCurator`. That module

- delegates lifecycle transitions and prompt rendering to `LemonSkills.Curator`

- and submits the review prompt to `LemonRouter` only when review is required.

-

- **Blueprint** is the portable distribution boundary for a coherent skill plus

- automation workflow. It reuses `LemonSkills.Bundle`, manifest/lint/audit, the

- derived `LemonCore.ProfileStore` workspace, and `CronManager`; do not add a

- second skill registry, profile store, or scheduler. A preview digest binds the

- manifest, content hashes, target profile, exact job projection, and current

- collision state. Activation replans under a lock, stages and re-audits exact

- copied bytes, enables profile-local skills, and calls `CronManager.add_new/1`

- last. The create-once manager/store path must never overwrite an existing

- stable ID. Version 1 remains agent-prompt-only and rejects commands, archives,

- symlinks, environment/cwd overrides, secret-like values, and non-UTC schedules.

- `LemonAutomation.Blueprint.Catalog` is the shared caller boundary for Web and

- control-plane clients: it derives the canonical catalog path from trusted

- profile options, accepts only bounded bundle IDs, rejects traversal and

- symlinked entries, enforces manifest/directory identity, and delegates all

- inspection, validation, preview, and activation semantics to `Blueprint`.

-

- ## Key Flow Details

-

- - `RunSubmitter` pre-subscribes to `Bus.run_topic(run_id)` BEFORE submitting to `LemonRouter` (avoids race condition)

- - `RunSubmitter` passes `run_id` in params so the router uses the same ID it already subscribed to

- - `RunSubmitter` executes each cron run in a forked `:sub:<id>` session for isolation; the originating base session is preserved in run metadata

- - `RunSubmitter` adds a cron-run tool policy with `blocked_tools: ["cron"]`; scheduled runs may do the task work, but they cannot recursively manage cron jobs through a cron tool

- - `RunSubmitter` reads `CronMemory` for the job and injects memory context into the prompt, then appends run results back to the memory file

- - If the router returns a different `run_id`, submission fails explicitly; no late replacement-id subscription is attempted

- - Output is truncated to 1000 chars before storage

- - Jobs execute in monitored `TaskSupervisor.async_nolink/2` tasks owned by `CronManager`; task-start failure and `:DOWN` terminalize the run immediately. Unsupervised fallback exists only when the manager is started with explicit `standalone_mode: true`.

- - For cron jobs created from `agent:*:main`, completion summaries are mirrored back into the base main session as synthetic `run_completed` entries (`meta.cron_forwarded_summary = true`)

- - Scheduled ticks and `CronManager` restarts consult persisted active runs before launching; active runs older than the job timeout recover through the same monitor/delivery/retry terminalizer as normal completions.

- - Scheduled runs use deterministic slot ids and `CronStore.claim_scheduled_run/3`, backed by `LemonCore.Store.put_new/3`, so competing dispatchers preserve the first claimant instead of overwriting a run.

- - Scheduled failures/timeouts persist retry due time, attempt, source, and root lineage on the terminal source run. Restart reconstructs the timer, while a deterministic `{root, attempt}` retry-run ID and atomic claim prevent duplicate attempts. Manual and wake runs stay single-shot by default.

- - Active cron runs can be aborted by cron run id. `CronManager.abort_run/1` calls the underlying router cancellation when possible, persists terminal `:aborted`, emits the normal completion event, and ignores late submitter completions.

- - Cron lifecycle actions write durable operator audit events to `:cron_audit_events`. The audit stream covers job create/update/pause/resume/delete, manual run requests, run start/abort/retry/stale recovery, and scheduled-run claim/suppression decisions. Audit entries keep operator-useful IDs in the store; support-bundle diagnostics redact those IDs.

- - `cron.status` reads the durable cron run and audit stores directly for operator-facing scheduler-health counters: active run locks, retry runs, suppressed scheduled slots, stale-run recoveries, scheduled retries, and next/last run timestamps.

- - Portable bundle activation must remain preview-first and content-free at the

- Web/control-plane boundary. Keep arbitrary paths local-only;

- `Blueprint.Catalog` resolves a safe `bundleId` below `~/.lemon/bundles`,

- returns no prompt/skill/path/secret text,

- and requires the exact fresh `confirmationDigest` for mutation.

- - Source and packaged `lemon blueprints` are thin clients of that RPC boundary.

- Do not move bundle loading or scheduler startup into the one-shot CLI VM;

- preview remains the bundle-ID shorthand and activation remains an explicit

- exact-digest request handled by the long-running `CronManager` runtime.

-

- ## Top-Level Facade

-

- `LemonAutomation` module provides delegating functions:

-

- ```elixir

- LemonAutomation.list_jobs() # delegates to CronManager.list/0

- LemonAutomation.add_job(params) # delegates to CronManager.add/1

- LemonAutomation.update_job(id, params) # delegates to CronManager.update/2

- LemonAutomation.remove_job(id) # delegates to CronManager.remove/1

- LemonAutomation.run_now(id) # delegates to CronManager.run_now/1

- LemonAutomation.abort_run(run_id) # delegates to CronManager.abort_run/1

- LemonAutomation.runs(job_id, opts) # delegates to CronManager.runs/2

- LemonAutomation.wake(job_id) # delegates to Wake.trigger/1

- ```

-

- ## How to Add a Cron Job

-

- ### Basic Job

-

- ```elixir

- {:ok, job} = LemonAutomation.CronManager.add(%{

- name: "Daily Report",

- schedule: "0 9 * * *", # 9 AM daily

- agent_id: "agent_abc",

- session_key: "agent:agent_abc:main",

- prompt: "Generate daily status report"

- })

- ```

-

- ### Command Job

-

- ```elixir

- {:ok, job} = LemonAutomation.CronManager.add(%{

- name: "Disk Usage Snapshot",

- schedule: "hourly",

- command: "df -h /",

- cwd: "/tmp"

- })

- ```

-

- ### With Timezone and Jitter

-

- ```elixir

- {:ok, job} = LemonAutomation.CronManager.add(%{

- name: "Hourly Check",

- schedule: "0 * * * *",

- agent_id: "agent_abc",

- session_key: "agent:agent_abc:main",

- prompt: "Check system status",

- timezone: "America/New_York", # Default: "UTC"

- jitter_sec: 30, # Random 0-30s delay before execution

- max_retries: 2, # Scheduled failure/timeout retries

- retry_backoff_ms: 60_000 # Delay before each retry

- })

- ```

-

- ### With Persistent Memory

-

- ```elixir

- {:ok, job} = LemonAutomation.CronManager.add(%{

- name: "Weekly Summary",

- schedule: "0 9 * * 1",

- agent_id: "agent_abc",

- session_key: "agent:agent_abc:main",

- prompt: "Generate weekly summary using notes from previous runs",

- memory_file: "/path/to/custom/memory.md" # Optional; auto-generated if omitted

- })

- ```

-

- ### Required Fields

-

- | Field | Description |

- |-------|-------------|

- | `name` | Human-readable identifier |

- | `schedule` | Cron expression (5 fields) or supported shorthand |

- | `agent_id` | Target agent for prompt jobs |

- | `session_key` | Routing key for prompt jobs (e.g., `"agent:{id}:main"`) |

- | `prompt` | Text sent to agent; mutually exclusive with `command` |

- | `command` | Local shell command for no-agent operator jobs |

-

- Updates preserve target type. Prompt jobs can update `prompt`; command jobs can

- update `command`, `cwd`, and `env`; `agent_id` and `session_key` stay immutable.

-

- ### Optional Fields

-

- | Field | Default | Description |

- |-------|---------|-------------|

- | `enabled` | `true` | Whether the job is active |

- | `timezone` | `"UTC"` | Timezone for schedule interpretation |

- | `jitter_sec` | `0` | Random delay spread in seconds |

- | `timeout_ms` | `300_000` | Max execution time (5 minutes) |

- | `max_retries` | `0` | Number of retries after scheduled failure/timeout |

- | `retry_backoff_ms` | `30_000` | Delay before retry runs |

- | `cwd` | runtime cwd | Command working directory |

- | `env` | `%{}` | Command environment overrides |

- | `memory_file` | auto | Path to persistent cross-run memory file |

- | `meta` | `nil` | Arbitrary metadata map |

-

- ### Cron Expression Format

-

- ```

- * * * * *

- | | | | +-- Day of week (0-7, 0/7 = Sunday)

- | | | +---- Month (1-12)

- | | +------ Day of month (1-31)

- | +-------- Hour (0-23)

- +---------- Minute (0-59)

- ```

-

- Supported cron syntax: `*`, `N`, `N-M` (range), `*/N` (step), `N,M,O` (list).

-

- Supported shorthands are normalized into 5-field cron expressions before

- storage. Interval shorthands must map cleanly to cron step fields, so minute

- intervals divide 60 and hour intervals divide 24:

-

- - `"every 30m"`, `"15 minutes"` -> `"*/30 * * * *"`, `"*/15 * * * *"`

- - `"hourly"`, `"every 2h"` -> `"0 * * * *"`, `"0 */2 * * *"`

- - `"daily at 9am"`, `"every day at 17:45"` -> `"0 9 * * *"`, `"45 17 * * *"`

- - `"weekdays at 09:30"` -> `"30 9 * * 1-5"`

- - `"weekly monday at 8am"`, `"fridays at 18:15"` -> `"0 8 * * 1"`, `"15 18 * * 5"`

-

- Examples:

- - `"*/15 * * * *"` - Every 15 minutes

- - `"0 9 * * 1"` - Mondays at 9 AM

- - `"0 */6 * * *"` - Every 6 hours

-

- ### Managing Jobs

-

- ```elixir

- # List all jobs (sorted by created_at_ms desc)

- LemonAutomation.CronManager.list()

-

- # Update a job (agent_id and session_key are immutable; attempts return {:error, {:immutable_fields, [...]}})

- LemonAutomation.CronManager.update(job.id, %{enabled: false})

-

- # Remove a job (also clears heartbeat config if job has meta.heartbeat: true)

- LemonAutomation.CronManager.remove(job.id)

-

- # Run immediately (manual trigger, job must exist but disabled jobs still work)

- LemonAutomation.CronManager.run_now(job.id)

-

- # Get run history

- # opts: limit (default 100), status (atom filter), since_ms (timestamp filter)

- LemonAutomation.CronManager.runs(job.id, limit: 10)

-

- # Force a tick cycle (for testing)

- LemonAutomation.CronManager.tick() # cast, returns :ok immediately

- ```

-

- ## CronJob Struct

-

- ```elixir

- %LemonAutomation.CronJob{

- id: "cron_abc123", # auto-generated via LemonCore.Id.cron_id()

- name: "Daily Report",

- schedule: "0 9 * * *",

- enabled: true,

- agent_id: "agent_abc",

- session_key: "agent:agent_abc:main",

- prompt: "Generate daily status report",

- memory_file: nil, # auto-generated path if nil

- timezone: "UTC",

- jitter_sec: 0,

- timeout_ms: 300_000,

- max_retries: 0,

- retry_backoff_ms: 30_000,

- created_at_ms: 1739989200000,

- updated_at_ms: 1739989200000,

- last_run_at_ms: nil,

- next_run_at_ms: 1739989200000,

- meta: nil

- }

- ```

-

- Key functions: `CronJob.new/1`, `CronJob.update/2`, `CronJob.due?/1`, `CronJob.mark_run/2`, `CronJob.set_next_run/2`, `CronJob.to_map/1`, `CronJob.from_map/1`.

-

- ## CronRun Struct and State Machine

-

- ```elixir

- %LemonAutomation.CronRun{

- id: "run_abc123", # auto-generated via LemonCore.Id.run_id()

- job_id: "cron_xyz789",

- run_id: nil, # LemonRouter run ID (set after submission)

- status: :pending, # :pending | :running | :completed | :failed | :timeout | :aborted

- triggered_by: :schedule, # :schedule | :manual | :wake | :retry

- started_at_ms: nil,

- completed_at_ms: nil,

- duration_ms: nil,

- output: nil, # truncated to 1000 chars

- error: nil,

- suppressed: false, # true if heartbeat with "HEARTBEAT_OK" response

- meta: nil

- }

- ```

-

- State transitions:

- ```

- CronRun.new(job_id, triggered_by) => status: :pending

- CronRun.start(run, run_id \\ nil) => status: :running

- CronRun.complete(run, output) => status: :completed

- CronRun.fail(run, error) => status: :failed

- CronRun.timeout(run) => status: :timeout

- CronRun.abort(run, reason) => status: :aborted

- CronRun.suppress(run) => suppressed: true (can combine with any terminal state)

- ```

-

- Helper predicates: `CronRun.active?/1` (pending or running), `CronRun.finished?/1`.

-

- Scheduled failures and timeouts retry only when `max_retries > 0`. The terminal

- source run persists `retry_due_at_ms`, `retry_next_attempt`, `retry_of`, and

- `retry_root_id`; each claimed retry is a separate run with

- `triggered_by: :retry`. Deterministic retry IDs make one root/attempt pair a

- stable uniqueness slot across manager restarts. Manual and wake-triggered runs

- do not retry by default.

-

- ## CronMemory (Persistent Cross-Run Memory)

-

- Each cron job can have a persistent markdown memory file that accumulates context across runs:

-

- - Default location: `~/.lemon/cron_memory/{job_id}.md`

- - Configurable via `memory_file` field on the job or `meta.memory_file`/`meta.memoryFile`

- - On each run, `RunSubmitter` reads the memory file and injects it into the prompt

- - After each run, the result is appended to the memory file

- - Auto-compaction kicks in at 24,000 chars: older content is summarized, recent 14,000 chars retained

- - Prompt injection limited to 8,000 chars (tail of file)

-

- ## Heartbeat Management

-

- Heartbeats are ordinary cron jobs; there is no heartbeat process, timer path,

- or separate store. `LemonAutomation.Heartbeat` supplies the policy that

- `CronManager` applies synchronously during run terminalization. Heartbeat

- schedules are cron schedules, so intervals must be cron-exact.

-

- ### Identifying Heartbeat Jobs

-

- A job is a heartbeat only when its `meta` has `heartbeat: true` (atom or string

- key). The job name is not consulted.

-

- ### Creating a Heartbeat via CronManager

-

- ```elixir

- {:ok, job} = LemonAutomation.CronManager.add(%{

- name: "heartbeat-agent_abc",

- schedule: "every 30m",

- timezone: "local",

- agent_id: "agent_abc",

- session_key: "agent:agent_abc:main",

- prompt: "Work through HEARTBEAT.md; reply HEARTBEAT_OK if nothing needs attention.",

- meta: %{heartbeat: true, agent_id: "agent_abc"}

- })

- ```

-

- The TUI `/heartbeat every <interval> <prompt>` command creates or updates this

- per-agent job (it ignores `meta.daily_brief` jobs).

-

- ### Heartbeat API

-

- ```elixir

- # meta.heartbeat == true (atom or string key)

- LemonAutomation.Heartbeat.heartbeat?(job)

-

- # Exact "HEARTBEAT_OK" after trimming

- LemonAutomation.Heartbeat.healthy_response?(response)

-

- # Called by CronManager terminalization; returns {run, deliver?}

- LemonAutomation.Heartbeat.apply_policy(job, run)

-

- # Daily brief: reconcile cron_daily_brief from [automation] daily_brief

- LemonAutomation.Heartbeat.sync_daily_brief()

- # ...or from an explicit HH:MM (nil removes the job); opts[:cron_manager] for tests

- LemonAutomation.Heartbeat.sync_daily_brief("08:00", cron_manager: MyCron)

- ```

-

- ### Heartbeat Suppression Behavior

-

- - Runs after the monitor and preflight policies in `CronManager.terminalize_run`

- - `:completed` + exact `HEARTBEAT_OK`: `CronRun.suppress/1`, kept in run history, emits `:heartbeat_suppressed`, NOT delivered

- - Any other outcome (including failure/timeout): emits `:heartbeat_alert` with `severity: :warning`, delivered through `CronDelivery`

- - Non-heartbeat jobs pass through unchanged

-

- ### Daily Brief

-

- `[automation] daily_brief = "HH:MM"` (`LemonCore.Config.Automation`) is

- reconciled once at boot by the `:daily_brief_sync` task into job

- `cron_daily_brief`: schedule `"M H * * *"`, `timezone: "local"`, agent

- `default`, session `agent:default:main`, `meta: %{heartbeat: true,

- daily_brief: true}`. A changed time updates schedule/timezone and re-enables

- the job; removing the key removes it; an invalid time is logged and returns

- `{:error, :invalid_time}` without touching jobs. `config :lemon_automation,

- daily_brief_sync: false` (used in `config/test.exs`) omits the task.

-

- ## Wake Functionality

-

- Wake triggers immediate job execution outside the normal schedule.

-

- ```elixir

- # Trigger single job (must be enabled, returns {:error, :job_disabled} otherwise)

- {:ok, run} = LemonAutomation.Wake.trigger("cron_abc123")

-

- # Trigger with context (stored in run.meta.wake_context)

- {:ok, run} = LemonAutomation.Wake.trigger("cron_abc123",

- context: %{reason: "incident response"}

- )

-

- # Skip if already running (checks CronStore.active_runs/1)

- {:ok, run} = LemonAutomation.Wake.trigger("cron_abc123",

- skip_if_running: true

- )

- # Returns {:error, :already_running} if active runs exist

-

- # Trigger multiple jobs (returns map of job_id => result)

- results = LemonAutomation.Wake.trigger_many(["cron_abc", "cron_xyz"])

-

- # Trigger by name pattern (case-insensitive substring match on enabled jobs)

- results = LemonAutomation.Wake.trigger_matching("heartbeat")

-

- # Trigger all enabled jobs for an agent

- results = LemonAutomation.Wake.trigger_for_agent("agent_abc")

- ```

-

- Wake runs use `triggered_by: :wake` and fire-and-forget: they return the

- `CronRun` immediately, use the same configured `:cron_run_submitter` as

- `CronManager`, and have completion handled asynchronously by `CronManager`.

- Wake failures do not enter the scheduled retry path.

-

- ## Events

-

- All events are broadcast on the `"cron"` bus topic as `%LemonCore.Event{}` structs.

-

- ### Event Types

-

- | Event | Emitted By | When |

- |-------|-----------|------|

- | `:cron_tick` | `CronManager` | Every 60s tick |

- | `:cron_job_created` | `CronManager` | Job added |

- | `:cron_job_updated` | `CronManager` | Job updated |

- | `:cron_job_deleted` | `CronManager` | Job removed |

- | `:cron_run_started` | `CronManager`, `Wake` | Run begins |

- | `:cron_run_completed` | `CronManager` | Run finishes (terminalization) |

- | `:cron_lifecycle_action` | `CronManager` | Durable audit event recorded |

- | `:heartbeat_suppressed` | `Heartbeat.apply_policy/2` (CronManager terminalization) | "HEARTBEAT_OK" response |

- | `:heartbeat_alert` | `Heartbeat.apply_policy/2` (CronManager terminalization) | Any other heartbeat outcome |

-

- ### Subscribing to Events

-

- ```elixir

- LemonCore.Bus.subscribe("cron")

-

- receive do

- %LemonCore.Event{type: :cron_run_started, payload: payload} ->

- IO.puts("Run started: #{payload.run.id}")

-

- %LemonCore.Event{type: :cron_run_completed, payload: payload} ->

- IO.puts("Run completed: #{payload.run.status}, suppressed: #{payload.suppressed}")

-

- %LemonCore.Event{type: :cron_lifecycle_action, payload: payload} ->

- IO.puts("Audit action: #{payload.audit.action}")

-

- %LemonCore.Event{type: :heartbeat_suppressed, payload: payload} ->

- IO.puts("Heartbeat OK for job #{payload.job_id}")

-

- %LemonCore.Event{type: :heartbeat_alert, payload: payload} ->

- IO.puts("Heartbeat alert for agent #{payload.agent_id}: #{payload.response}")

- end

- ```

-

- ## CronStore API

-

- ```elixir

- # Jobs (tables: :cron_jobs)

- CronStore.put_job(job)

- CronStore.get_job(job_id)

- CronStore.delete_job(job_id)

- CronStore.list_jobs() # all jobs, sorted by created_at_ms desc

- CronStore.list_enabled_jobs() # only enabled: true

- CronStore.list_due_jobs() # enabled and due?(job) == true

-

- # Runs (table: :cron_runs)

- CronStore.put_run(run)

- CronStore.claim_run(run)

- CronStore.claim_scheduled_run(job, scheduled_for_ms, router_run_id)

- CronStore.claim_retry_run(job, source_run, router_run_id)

- CronStore.retry_run_id(root_run_id, attempt)

- CronStore.pending_retries()

- CronStore.get_run(run_id)

- CronStore.delete_run(run_id)

- CronStore.list_runs(job_id, opts) # opts: limit (100), status (atom), since_ms

- CronStore.list_all_runs(opts) # across all jobs

- CronStore.active_runs(job_id) # runs where status in [:pending, :running]

- CronStore.cleanup_old_runs(keep_per_job \\ 100)

-

- # Audit (table: :cron_audit_events)

- CronStore.record_audit(action, attrs)

- CronStore.list_audit_events(opts) # opts: limit, job_id, run_id, action, since_ms

- CronStore.delete_audit_event(id)

- ```

-

- ## CronSchedule API

-

- ```elixir

- # Parse expression into structured map

- {:ok, parsed} = CronSchedule.parse("*/15 * * * *")

- # parsed = %{minute: [0,15,30,45], hour: [0..23], day: [1..31], month: [1..12], weekday: [0..6]}

-

- # Get next run timestamp in milliseconds

- CronSchedule.next_run_ms("0 9 * * *", "UTC") # => integer ms | nil

- CronSchedule.next_run_datetime("0 9 * * *", "UTC") # => %DateTime{} | nil

-

- # Get multiple future run times

- CronSchedule.next_runs("*/15 * * * *", "UTC", count: 5) # => [%DateTime{}, ...]

-

- # Check if a DateTime matches an expression

- CronSchedule.matches?("0 9 * * *", datetime) # => boolean

-

- # Validate expression

- CronSchedule.valid?("0 9 * * *") # => boolean

- ```

-

- ## Common Tasks

-

- ### Check Job Status

-

- ```elixir

- alias LemonAutomation.{CronStore, CronJob}

-

- # Check if job is currently due

- job = CronStore.get_job("cron_abc123")

- CronJob.due?(job)

-

- # Get active runs

- active = CronStore.active_runs("cron_abc123")

-

- # Get last run

- [last | _] = CronStore.list_runs("cron_abc123", limit: 1)

- ```

-

- ### Clean Up Old Runs

-

- ```elixir

- # Keep only last 50 runs per job (default: 100)

- LemonAutomation.CronStore.cleanup_old_runs(50)

- ```

-

- ## Testing Guidance

-

- ### Test Structure

-

- ```

- test/lemon_automation/

- +-- cron_job_test.exs # CronJob struct lifecycle

- +-- cron_run_test.exs # CronRun state transitions

- +-- cron_schedule_test.exs # Cron parsing, next_run computation

- +-- cron_store_test.exs # Persistence operations

- +-- cron_manager_update_test.exs # Immutable field rejection

- +-- cron_manager_forwarding_test.exs # Base-session forwarding, owner-chat default, deliver_to, heartbeat suppression, one-shot removal

- +-- cron_delivery_test.exs # deliver_to validation, target resolution, channel text

- +-- events_test.exs # Event emission

- +-- heartbeat_test.exs # apply_policy/2 and daily-brief reconciliation

- +-- run_completion_waiter_test.exs # Wait logic

- +-- run_submitter_test.exs # Router submission

- +-- goal_loop_test.exs # Goal loops and router judge proof

- +-- goal_judge_router_live_test.exs # Opt-in provider-backed judge proof

- +-- kanban_dispatcher_test.exs # Dispatcher leasing/concurrency/failure proof

- +-- kanban_dispatcher_live_test.exs # Opt-in provider-backed dispatcher proof

- +-- kanban_run_worker_test.exs # Router-backed kanban worker proof

- +-- wake_test.exs # Wake triggering

- ```

-

- ### Running Tests

-

- ```bash

- # All automation tests

- mix test apps/lemon_automation

-

- # Specific module

- mix test apps/lemon_automation/test/lemon_automation/cron_schedule_test.exs

-

- # Provider-backed goal-judge proof

- ZAI_API_KEY="$(MIX_ENV=dev mix run --no-start -e 'Logger.configure(level: :emergency); Logger.remove_backend(:console); {:ok, _} = Application.ensure_all_started(:lemon_core); IO.write(LemonCore.Secrets.fetch_value("llm_zai_api_key") || "")')" \

- LEMON_TEST_ALLOW_LIVE_CREDENTIALS=1 \

- LEMON_GOAL_JUDGE_MODEL="zai:glm-5-turbo" \

- scripts/test path apps/lemon_automation/test/lemon_automation/goal_judge_router_live_test.exs --include integration --seed 1

-

- # Provider-backed kanban dispatcher proof

- secret_file=$(mktemp "${TMPDIR:-/tmp}/lemon-zai-secret.XXXXXX")

- LEMON_SECRET_OUTPUT="$secret_file" MIX_ENV=dev mix run --no-start -e '

- Logger.configure(level: :emergency)

- Logger.remove_backend(:console)

- {:ok, _} = Application.ensure_all_started(:lemon_core)

- case LemonCore.Secrets.fetch_value("llm_zai_api_key") do

- value when is_binary(value) and value != "" -> File.write!(System.fetch_env!("LEMON_SECRET_OUTPUT"), value)

- _ -> System.halt(66)

- end

- ' >/dev/null 2>&1

- ZAI_API_KEY="$(cat "$secret_file")" \

- LEMON_TEST_ALLOW_LIVE_CREDENTIALS=1 \

- LEMON_KANBAN_LIVE_MODEL="zai:glm-5-turbo" \

- scripts/test path apps/lemon_automation/test/lemon_automation/kanban_dispatcher_live_test.exs --include integration --seed 1

- rc=$?

- rm -f "$secret_file"

- exit "$rc"

-

- # With coverage

- mix test --cover apps/lemon_automation

-

- # Portable bundle policy, rollback, exact confirmation, and replay

- mix test apps/lemon_automation/test/lemon_automation/blueprint_test.exs --seed 1

-

- ```

-

- ### Key Testing Patterns

-

- **Mock the Router:**

-

- ```elixir

- test "submits to router", %{job: job, run: run} do

- defmodule MockRouter do

- def submit(params) do

- send(self(), {:submitted, params})

- {:ok, "run_123"}

- end

- end

-

- result = RunSubmitter.submit(job, run,

- router_mod: MockRouter,

- waiter_mod: MockWaiter

- )

- end

- ```

- **Test Cron Parsing:**

- ```elixir

- test "parses valid cron expressions" do

- assert {:ok, parsed} = CronSchedule.parse("*/15 * * * *")

- assert parsed.minute == [0, 15, 30, 45]

- end

- ```

- **Test Heartbeat Suppression:**

- ```elixir

- test "suppresses exact HEARTBEAT_OK" do

- # Exact match and whitespace-trimmed variants are suppressed

- assert Heartbeat.healthy_response?("HEARTBEAT_OK")

- assert Heartbeat.healthy_response?(" HEARTBEAT_OK ")

- assert Heartbeat.healthy_response?("HEARTBEAT_OK\n")

- # Any other string is NOT suppressed

- refute Heartbeat.healthy_response?("Status: OK")

- refute Heartbeat.healthy_response?("HEARTBEAT_OK extra")

- refute Heartbeat.healthy_response?(nil)

- end

- ```

- **Test State Transitions:**

- ```elixir

- test "run state transitions" do

- run = CronRun.new("job_1", :schedule)

- assert run.status == :pending

- run = CronRun.start(run)

- assert run.status == :running

- run = CronRun.complete(run, "output")

- assert run.status == :completed

- run = CronRun.suppress(run)

- assert run.suppressed == true

- end

- ```

- **Force a Tick:**

- ```elixir

- # Cast-based tick (returns immediately, execution is async)

- LemonAutomation.CronManager.tick()

- ```

- ## Dependencies

- - `lemon_core` - Store, Bus, Clock, Id generation

- - `lemon_router` - Run submission via `LemonRouter.submit/1`

- - `jason` - JSON serialization

- ## Key Modules Reference

- | Module | Purpose |

- |--------|---------|

- | `LemonAutomation` | Top-level facade with delegating functions |

- | `LemonAutomation.Application` | OTP supervisor (TaskSupervisor, CronManager, daily-brief sync task, goal/kanban/curator/synthesis managers) |

- | `LemonAutomation.Blueprint` | Portable manifest validation, exact preview confirmation, profile-local skill activation, and idempotent cron claim |

- | `LemonAutomation.CronManager` | Scheduling GenServer; owns job state in-memory + persists to CronStore |

- | `LemonAutomation.CronJob` | Job struct, CRUD ops, `due?/1` predicate |

- | `LemonAutomation.CronRun` | Run struct, state machine transitions |

- | `LemonAutomation.CronSchedule` | Cron parsing, next-run computation, `valid?/1`, `matches?/2` |

- | `LemonAutomation.CronStore` | Persistence via LemonCore.Store (tables: `:cron_jobs`, `:cron_runs`) |

- | `LemonAutomation.CronMemory` | Persistent markdown-based cross-run memory; auto-compaction |

- | `LemonAutomation.Heartbeat` | Heartbeat policy (`meta.heartbeat`, `HEARTBEAT_OK` suppression) applied in CronManager terminalization; daily-brief reconciliation |

- | `LemonAutomation.CronDelivery` | Delivery target resolution (`deliver_to`, originating chat, paired owner chat), channel text, outbox enqueue |

- | `LemonAutomation.SkillCurator` | Idle/config gates plus background submission for learned-skill curator prompts with a learning-only default tool policy |

- | `LemonAutomation.SkillCuratorManager` | Periodic idle scheduler for `SkillCurator.run_once/1` |

- | `LemonAutomation.Wake` | Manual immediate triggering (fire-and-forget, enabled jobs only) |

- | `LemonAutomation.RunCompletionWaiter` | Owns race-free fixed-id submit/wait subscriptions and parses terminal events |

- | `LemonAutomation.RunSubmitter` | Builds params, pre-subscribes to bus, submits to LemonRouter, manages CronMemory |

- | `LemonAutomation.Events` | Event emission helpers for all automation events |