standards_and_architecture · diff
git:20260905.b17d431 to git:20260906.f9cef19
480 added, 227 removed. Audit A to A.
---
name: standards_and_architecture
description: The definitive source of truth for all coding conventions, environment constraints, and architecture layouts in the SICTIC-AI repository. The AI must review this skill before writing, modifying, or refactoring any Python code to ensure strict adherence.
---
# Standards and Architecture
- This skill acts as a mandatory pre-flight checklist. Review these standards before writing or modifying any Python code.
+ This skill defines repository technical contracts. Read it before modifying code.
- ## Data Storage Layout (`LOCAL_STORAGE_PATH`)
+ Follow the repository-root [AGENTS.md](../../AGENTS.md) for scope,
+ approval, implementation review and verification procedures.
- The AI and all skills must strictly adhere to the configured storage-domain
- layout. The local root is defined by `LOCAL_STORAGE_PATH`; general application
- storage is always local. Optional external synchronization must not be called
- from skill code. The domain roots are defined in
- `config/storage_domains.json`; code must use `lib.datasets.paths` rather than
- hardcoding storage paths.
+ See [Architecture and layout reference](references/architecture.md)
+ for the storage layout, module map and skill-directory example.
- * `config/` — A folder structure filled with `.md` and `.json` files containing prompts and settings. The infrastructure configuration module compiles these into a single configuration tree at runtime.
- * `storage/startups/<dataset_name>/datasets/` — raw startup data rooms.
- * `__active_dataset__.md` — A readable marker file. If present, it signals to batch jobs (like `bulk_refresh`) that this startup should be actively processed.
- * `storage/startups/<dataset_name>/insights/` — generated startup AI reports and profiles.
- * `storage/community/<dataset_name>/datasets/` — primary community/member datasets.
- * `storage/community/<dataset_name>/insights/` — generated community AI reports and profiles.
- * `storage/generated/<dataset_name>/datasets/` — searchable datasets assembled from insights, such as member profiles used by ranking.
- * `storage/generated/<dataset_name>/insights/` — insights associated with generated datasets, when applicable.
- * `docling_data/datasets2md/<domain>/<dataset_name>/datasets/` — durable machine-local Docling-extracted Markdown storage. This is generated, but it is not disposable cache and is not part of the synchronized application-storage tree.
- * `storage/<domain>/<dataset_name>/insights/persons-in-dataset-<dataset_name>-manual.md` — manually maintained person lists generated by `persons_in_dataset`. Once created, these files are treated as user-edited source-of-truth inputs and are never overwritten.
- * `cache/` — disposable runtime cache and temporary operational state.
+ ## Environment and common conventions
- *(Note on Data Sharing: If a Deal Lead needs access to a specific startup's data, do not break this structure. Share or synchronize a view without changing the configured hierarchy used by the AI.)*
+ - Execute Python through the `sictic-env` Conda environment, bootstrapped
+ from `environment.yml` by `install.sh`.
+ - The installer adds the repository root through a `.pth` file, so
+ `skills` and `lib` imports work from any CWD without `PYTHONPATH`.
+ - Python files and skill packages use lowercase snake_case names.
+ Dataset names are lowercase.
+ - Functions instantiate single-use dependencies unless sharing persistent
+ state such as a database connection pool. Preserve existing optional client
+ and storage injection points for testing and caller-owned resources.
- ## Coding Standards
+ ## People: identity, evidence and workflows
- * **Parameter Order:** If `dataset` (or `dataset_name`) is a parameter for a routine, function, or skill API, it must **always** be the first parameter.
- * **Refactoring Protocol:**
- * Before refactoring any code, you must first ask clarifying questions to the user.
- * After getting answers, propose the exact refactoring approach/plan.
- * **Only** after the user explicitly agrees to the proposed approach are you allowed to execute the refactor and edit the code.
- * **Testing Protocol:**
- * If you write or use temporary Python scripts to test or verify functionality in the codebase, you must always ask the user afterwards if that script should be converted into a formal `pytest` unit/integration test.
- * **Environment:** All code is executed via the `sictic-env` Conda environment, bootstrapped from `environment.yml` by `install.sh`.
- * **Python Path:** The installer writes the repository root to a `.pth` file in the Conda environment, so `import skills.<SKILL_NAME>.<SCRIPT_NAME>` and `import lib.<MODULE>` resolve from any CWD without setting `PYTHONPATH`.
- * **Imports:** User-facing skill code lives under `skills.<SKILL_NAME>.<SCRIPT_NAME>`. Shared infrastructure lives under `lib.<MODULE>` (logger, env, adapters, slugify, etc.).
- * **Naming Conventions:**
- * All skill directory names must only contain underscores (`_`). **Never** hyphens (`-`).
- * All Python files (`.py`) must be strictly lowercase with underscores (snake_case). No uppercase letters (e.g., `config_load.py`, not `Config_Load.py`).
- * Dataset names are always strictly `lowercase`.
- * **Data Structures (Person Wrapper):** Whenever passing person data through LinkedIn resolution and downstream skills, use the standard `lib.people.model.Person` wrapper:
- ```python
- Person(
- full_name=str, # Sanitized: Latin characters preserved, emojis/symbols stripped
- linkedin_id=str, # Canonical slug (e.g., 'john-doe'). This is the unique ID and cache filename
- email_addresses=list, # Normalized lowercase email addresses for matching and contact tables
- linkedin_profile=dict, # Full JSON payload returned by the scraper
- dossier=list, # High-value complete Chunk documents (e.g., full resumes/CVs text)
- mentions=list, # Isolated Chunk mentions where the person appears downstream
- person_profile_markdown=str, # Generated Markdown profile when available
- )
- ```
- * **Output Standards:**
- * All output files generated by skills must be saved under the configured `insights_root` for the dataset's storage domain.
- * All output filenames must end with the slugified name of the model used (e.g., `-<MODEL_NAME>.md`). Note: `<MODEL_NAME>` strictly refers to the model part without the provider prefix, normalized by `lib.slugify` (e.g., `ollama/qwen3.5:4b` becomes `qwen3.5-4b`).
- * **File Naming Convention:** All generated user-facing output files (like Markdown reports) must strictly follow `kebab-case`. The script must build a raw filename string and pass it through the global `slugify()` utility (found in `lib.slugify`) to guarantee uniform sanitization of spaces, underscores, and accents before appending the `.md` extension. System identifiers (like dictionary keys, dataset IDs, and `.py` module names) should remain in `snake_case`.
- * **Logging vs. Printing:**
- * Extensive logging must be implemented using the centralized logger utility.
- * Every script must import it at the top level: `from lib.infrastructure.logging import get_logger` followed by `logger = get_logger(__name__)`.
- * This ensures all logs, across all skills and levels (INFO, WARNING, ERROR, DEBUG), flow into the single `{{REPO_ROOT}}/logs/sictic-ai.log` file chronologically.
- * `print()` and `rich` consoles are reserved **strictly** for final output delivery to the user/GUI. All internal state, progress steps, and warnings must use the `logger`.
- * **CLI/Routing:**
- * `typer` is used for function descriptions, CLI routing, and argument parsing.
- * Every executable skill must have a clearly defined Typer entry point.
- * **Thin CLI (Separation of Concerns):** Typer entry points (`__main__.py`) must contain **zero** business logic. They only parse arguments, handle top-level exception catching, and pass execution to core engine modules.
- * **Configuration & Prompts:** Hardcoding prompts inside Python files is forbidden. All prompts, instructions, and tuning parameters must be stored under the repository's `config/` folder and fetched dynamically via `load_repository_config(*sections)`. Never use default values when sourcing required configuration; use direct key access so missing configuration fails clearly.
- * **Encapsulated Dependencies:** Functions should instantiate their own single-use dependencies (like Docling/Rclone for ingestion) rather than forcing parent functions to instantiate and pass them down, unless injecting a shared persistent state like a database connection pool.
- * **Error Handling & Boundaries:**
- * Strict error reporting is mandatory.
- * Utility modules, adapters, and core functions must **never** call `sys.exit()` or `raise typer.Exit()`. They must raise standard Python Exceptions (e.g., `ValueError`, `RuntimeError`).
- * Only the top-level CLI wrapper (`@app.command()`) is allowed to catch these exceptions, log them, and execute a graceful `raise typer.Exit(code=1)`.
- * **LLM Inference (LiteLLM):** All LLM inferences (both Text Generation and Embeddings) MUST be routed through the `litellm` library. Direct API calls to specific providers (like Ollama, OpenAI, Anthropic) are strictly forbidden for inference.
+ ### Canonical person object
- ## Architecture
+ Use [lib.people.model.Person](../../lib/people/model.py) between discovery,
+ enrichment, profiling and consumers. Preserve identity, contact, evidence and
+ profile fields; table-specific values belong in namespaced `adhoc_data`.
- Two top-level Python packages: `skills/` (user-facing CLI skills, each with `SKILL.md` + `__main__.py`) and `lib/` (shared infrastructure, no `SKILL.md`).
+ `Person` normalizes LinkedIn IDs and emails on construction. Reuse its module's
+ `normalize_email_addresses` and `extract_email_addresses`. Full-name cleanup
+ belongs to LinkedIn resolution's `sanitize_name`, not the constructor.
- ### Architectural Layers
+ ### Identity and matching
- `lib/` is no longer a flat utility folder. It is organized around storage,
- dataset ingestion, insight lifecycle, and business domains. Skills should
- compose these shared libraries rather than duplicating path, cache, identity, or
- freshness logic.
+ Use `Person.identifier`: **LinkedIn ID → first normalized email → slugified
+ full name**. `Person.display_name` is display text, not artifact identity.
+ Use `extract_linkedin_id` from `lib.people.linkedin` for URLs or ID inputs.
- 1. **Runtime and Storage Foundation**
- * `lib.infrastructure.configuration`: Repository configuration, environment access and `.env` loading.
- * `lib.infrastructure.logging`: Centralized logging to `logs/sictic-ai.log`.
- * `lib.model_config`: Runtime model and endpoint configuration.
- * `lib.infrastructure.scheduler`: Cross-process scheduling for local-service
- concurrency and cloud token limits. Its shared state lives at
- `<LOCAL_DATA_PATH>/cache/scheduler.json`.
- * Model, embedding, reranking, and document-conversion modules schedule
- provider operations through `lib.infrastructure.scheduler`.
- * `lib.slugify`: Canonical filename and identifier slugification.
- * `lib.storage`: Relative-path storage abstraction. Application data is
- routed to `LOCAL_STORAGE_PATH`; machine-local runtime data under `cache/`
- and `docling_data/` is routed to `LOCAL_DATA_PATH`/`REPO_PATH`.
- * `lib.infrastructure.dealum`: Provider-specific Dealum API access
- * `lib.infrastructure.qdrant`: Dataset-scoped Qdrant storage and database
- administration
- * `lib.infrastructure.web_search`: Provider-specific web search access
- backed by Apify. Infrastructure adapters must not own business-domain
- decisions.
+ Reconcile through `Person.match_score`, `matches`, `find_best_match` and `merge`.
+ Confirm the match before merging; `merge` does not validate compatibility.
+ LinkedIn-cache resolution uses the stricter `find_cached_person` policy and
+ preserves explicit ID boundaries. Do not substitute general matching for it.
+ Preserve existing thresholds and workflow-specific rules for supplied contacts.
- 2. **Dataset Domain (`lib.datasets`)**
- * Owns dataset discovery, storage-domain path resolution, active/archive
- markers, source conversion, chunking, embedding, Qdrant indexing, and
- semantic search.
- * `paths.py` is the only supported API for dataset locations. It reads
- `config/storage_domains.json` and exposes `DatasetLocation`,
- `dataset_location(...)`, `dataset_location_for_domain(...)`,
- `dataset_raw_path(...)`, `dataset_parsed_path(...)`,
- `dataset_insights_path(...)`, and dataset listing helpers.
- * `source.py` discovers ingestible files and ignores non-text/binary/source
- control files.
- * `conversion.py` reconciles raw source files into durable parsed Markdown
- using Docling.
- * `indexing.py` chunks parsed Markdown, embeds chunks through LiteLLM, and
- reconciles complete document replacements in Qdrant.
- * `manifest.py` stores conversion/indexing freshness state and the indexed
- dataset revision used by insight freshness checks.
- * `ingestion.py` orchestrates conversion and indexing through
- `sync_datasets(...)`.
- * `search.py` exposes `dataset_search(...)`; callers must treat it as a
- side-effecting operation because it synchronizes the dataset before
- querying Qdrant.
+ ### Authoritative roster and table parsing
- 3. **Insight Domain (`lib.insights`)**
- * Owns generated insight paths, filenames, model suffixes, freshness
- manifests, model fallback selection, manual overrides, and generated
- dataset hydration.
- * Use `lib.insights.InsightFile` for generated Markdown reports. Do not
- hardcode insight paths and do not use legacy `insight_filepath` or
- `insight_refresh` patterns.
- * `InsightFile.find(selection="reusable")` selects a fresh insight by checking manual
- overrides first, then ranked models whose prompt hash and source dataset
- revisions match the current state.
- * `InsightFile.find(selection="any")` selects the best available existing insight when
- strict freshness is not required.
- * `InsightFile.find_all(skill=..., datasets=..., selection="any")` discovers
- stored logical insights and returns one selected `InsightFile` for each.
- Bulk reusable selection is not supported because the expected prompt and
- source datasets are not available during discovery.
- * `hydration.py` builds generated datasets from selected insight files, for
- example `storage/generated/sictic-members-investor-profile/datasets/`.
+ The [persons_in_dataset skill](../persons_in_dataset/SKILL.md) owns discovery
+ and initial roster creation. [lib.people.discovery](../../lib/people/discovery.py)
+ owns reading and rendering. Its synchronous readers never discover or enrich:
- 4. **Startup Domain (`lib.startups`)**
- * Owns startup identity, standard startup dossier layout, active startup
- dataset creation, and external startup source hydration.
- * `identity.py` resolves canonical startup slugs and configured aliases.
- * `dossier.py` creates the standard startup raw and parsed subdirectories
- (`data-room`, `linkedin`, `dealum`, `snippets`, `post-deal`) and active
- markers.
- * `sources.py` exposes `ensure_startup_dataset(...)`, which verifies local
- startup data and optionally refreshes it from Dealum.
- * `startups/dealum/` owns Dealum reconciliation, exact name/application-code
- matching, application Markdown rendering, manifest hashing, document
- downloads, and import results.
+ - `manual_persons_in_dataset`: absent file returns `None`.
+ - `persons_in_dataset`: absent file raises, directing callers to the skill.
+ - Both return `[]` for an intentionally empty valid table and reject invalid input.
- 5. **People Domain (`lib.people`)**
- * Owns the canonical `Person` model, person identity matching/merging,
- editable person roster parsing, and person dossier assembly.
- * `model.py` defines the `Person` dataclass and email/name matching helpers.
- * `discovery.py` reads and renders the manual persons roster. Its synchronous
- `persons_in_dataset(...)` compatibility reader requires an existing roster;
- it performs no discovery or enrichment. The async `persons_in_dataset`
- skill owns discovery and creation of the editable manual insight.
- * `dossier.py` builds a person's document dossier and incidental mentions
- from parsed dataset content and semantic search.
- * `linkedin/` owns LinkedIn identifier parsing, dataset-local profile files,
- the outstanding-profile registry, Apify job reconciliation, manual
- imports, technical payload cleaning, and `Person` enrichment. Stored
- profiles exclude web links and low-signal provider fields before dataset
- indexing. Only generic Apify access remains in `lib.infrastructure.apify`.
+ The manual roster is authoritative. Preserve edits; generated discovery JSON
+ is not an alternative input. Empty discovery must not create a permanent empty
+ roster. Consumers must read it without implicit discovery; bulk refresh may run
+ the declared discovery dependency.
+ The discovery sources and `sictic-members` cache-only path belong in the skill.
- 6. **Small Utilities**
- * `lib.insights.select_insights(source_datasets, skill)` returns the
- preferred stored file for every logical insight without materializing or
- ingesting a dataset.
- * `lib.insights.dataset_from_insight(target_dataset, source_datasets,
- skill)` uses that selection to reconcile a caller-named generated dataset
- and returns the selected `list[InsightFile]`.
- * `lib.ephemeral_dataset` prepares temporary generated datasets in Qdrant.
- * `lib.json_parser`, `lib.litellm_cleanup`, and `lib.runtime_noise` are small
- cross-cutting helpers.
+ Use the roster reader for roster inputs. For other person tables, use
+ [markdown_table_to_person_objects](../../lib/people/markdown.py) and preserve
+ additional columns under their tag.
- ### `skills/` — user-facing CLI skills
+ ### Person evidence gathering
- Skills are user-facing orchestration packages. A skill may call other skills
- when it is composing user-facing workflows, but reusable infrastructure belongs
- in `lib/`.
+ Use [build_person_dossier](../../lib/people/dossier.py) for separate dossier and
+ incidental-mention lists. Its `get_filtered_chunks`, `person_in_filename` and
+ `is_personal_document` own retrieval filtering and document selection.
+ Dossiers exclude LinkedIn documents; profiling supplies that evidence separately.
+ Preserve source-document and page metadata; do not recreate selection in skills.
- * `lib.infrastructure.configuration`: Compiles repository configuration into runtime prompts and settings
- dictionaries.
- * `dataset_chat`: User-facing RAG question answering over datasets. Shared
- ingestion/search code belongs in `lib.datasets`.
- * `dataset_maintenance`: Operational diagnostics, migration, pruning, and
- generated-dataset hydration commands.
- * `linkedin_maintenance`: Human-in-the-loop tooling for missing profiles,
- manual imports, and registry diagnosis.
- * `persons_in_dataset`: Reads the authoritative manual roster first; when absent,
- discovers related names from indexed data-room evidence without requiring
- LinkedIn or public search, and retains explicit in-dataset LinkedIn IDs.
- `person_profile` and downstream consumers only read the existing roster;
- discovery runs explicitly or as a bulk-refresh dependency.
- * `person_profile`, `investor_profile`, `startup_profile`, `team_profile`,
- `startup_traction`, `dd_checks`: Insight-producing profile and due-diligence
- skills. They must save reports through `InsightFile` and their primary Python
- APIs must return a flat `list[InsightFile]`, including cached results.
- * `ranking`: Shared ranking skill package used by matchmaker-style skills.
- Its `ranking_top_k.py`, `ranking_rationale.py`, and `ranking_persons.py`
- live under `skills/ranking`, not `lib/`.
- * `expert_search`, `potential_investors`, `advocates`, `suggested_startups`:
- Matching workflows that select stored investor or startup profiles and rank
- their contents directly without generated-dataset ingestion. Their primary
- Python APIs also return a flat `list[InsightFile]`.
- * `bulk_refresh`: Batch orchestration across active datasets and skills.
- * `dealum_import`: CLI wrapper around `lib.startups.dealum`.
+ ### LinkedIn retrieval and persistence
- ## Skill Directory Structure
+ Use [LinkedInResolver](../../lib/people/linkedin/service.py):
- Every skill must strictly adhere to the following Python package structure. Custom subdirectories like `scripts/` are forbidden.
+ - `get_cached_persons`: cached `Person` objects without external requests.
+ - `get_profiles`: enriches people; may fetch profiles, write stored files and
+ registry state, or raise unresolved-profile errors.
+ - `get_profiles(allow_scrape=False)`: can still register missing profiles;
+ it is not read-only.
+ - `get_all_persons`: display-name strings, not a roster or complete objects.
- ```text
- {{REPO_ROOT}}/
- ├── lib/ # GLOBAL: Shared infrastructure (env, logger, adapters, …)
- │
- ├── skills/
- │ └── <SKILL_NAME>/ # The individual skill package
- │ ├── SKILL.md # Mandatory: AI instructions and metadata for the skill
- │ ├── __init__.py # Makes the skill directory a Python package
- │ ├── __main__.py # Thin CLI Entry Point (Typer app)
- │ ├── <skill_name>.py # Primary Class/API facade - Logic Entry Point
- │ ├── core/ # (Optional) Subroutines and heavy business logic
- │ │ └── __init__.py
- │ └── utils/ # (Optional) Skill-specific stateless helpers
- │ └── __init__.py
- │
- └── tests/ # GLOBAL: pytest suite
- ```
+ Use [LinkedInProfileStore](../../lib/people/linkedin/store.py); its `write`
+ applies shared `clean_linkedin_payload`. Stored LinkedIn JSON and generated
+ profile Markdown are distinct artifacts. Do not vary stored payloads by skill.
- * **`SKILL.md`:** This file is mandatory and must always be present in the root of the skill directory. It contains the prompt instructions and metadata that OpenClaw, Nemoclaw, Claude, and other harnesses use to understand and trigger the skill. For user-facing skills exposed by the harness, the `## Usage` section should provide a universal, copy-pastable slash command formatted as: `conda run -n sictic-env python -m skills.harness /<COMMAND> [args]`.
- * **`__main__.py`:** Contains zero business logic. Handles Typer CLI routing, argument parsing, and top-level exception catching. Allows execution via `python -m skills.<SKILL_NAME>`. This is the *only* allowed Typer CLI entry point for a skill.
- * **`<skill_name>.py`:** The primary programmatic entry point (orchestrator/facade) for the skill.
- * **Function Naming Rule:** The main API function inside this file must be named identically to the skill itself (e.g., `def startup_profile(...):` inside `startup_profile.py`). External modules must call this specific function.
- * **Parameter Standardization:** When applicable, the primary API function of a skill must always take `dataset_name: str` as its very first parameter. This ensures consistency for automated batch runners like `bulk_refresh`.
- * **`core/` & `utils/` (Optional):** These directories should only be generated if the skill is complex enough to require them.
- * **`core/`:** Contains internal subroutines and heavy lifting.
- * **Local `utils/`:** Contains stateless helper functions strictly unique to this specific skill. (For cross-skill shared infrastructure, use the top-level `lib/` directory instead.)
+ Use [LinkedInRegistry](../../lib/people/linkedin/registry.py) mutation methods
+ and their locked updates, not independent load–modify–save or direct JSON edits.
+ Use `linkedin_profile_not_found` to distinguish confirmed absence from failed or
+ pending retrieval. [diagnose_registry and import_profiles](../../lib/people/linkedin/maintenance.py)
+ own maintenance; copying JSON alone does not reconcile stored profiles and registry.
+
+ ### Standard person profiles
+
+ [person_profile](../person_profile/SKILL.md) reads the roster, enriches LinkedIn
+ and gathers dataset evidence. Its standard prompt includes founder-trait
+ instructions; assess explicitly identified active founders and state insufficient
+ information where unsupported. Other people remain factual profiles.
+
+ `person_profile` returns `list[InsightFile]`; `person_profile_as_person_objects`
+ returns populated `list[Person]` from the same workflow. Explicit-name selection
+ and its current out-of-roster behavior are documented in that skill.
+
+ Pass `Person.identifier` to `InsightFile`: `<identifier>-<model>.md` with shared
+ filename normalization. Preserve manual profile precedence and the existing
+ `include_dataset_context` option. Do not add assessment or workflow suffixes.
+ Registry and direct team calls use the same standard profile settings and caches.
+
+ ### Skill responsibilities
+
+ Use these skill documents for selection rules, prompts, CLI options and limitations:
+
+ | Owner | People workflow |
+ |---|---|
+ | [persons_in_dataset](../persons_in_dataset/SKILL.md) | Discovery and editable-roster creation. |
+ | [person_profile](../person_profile/SKILL.md) | Enrichment and standard individual profiles. |
+ | [linkedin_maintenance](../linkedin_maintenance/SKILL.md) | Missing-profile diagnostics and manual imports. |
+ | [member_preferences](../member_preferences/SKILL.md) | Roster identities and namespaced preferences. |
+ | [investor_profile](../investor_profile/SKILL.md) | Stored profiles plus manual track records/preferences. |
+ | [team_profile](../team_profile/SKILL.md), [team_profile_revised](../team_profile_revised/SKILL.md) | Team assessment using standard profiles. |
+ | [ranking](../ranking/SKILL.md), [suggested_startups](../suggested_startups/SKILL.md) | Selection from existing identities and profiles. |
+ | [deep_dive_invitation](../deep_dive_invitation/SKILL.md) | Supplied contacts, preferences and reconciliation. |
+ | [bulk_refresh](../bulk_refresh/SKILL.md) | Declared discovery/profiling dependencies. |
+
+ Do not assume all consumers support every identity fallback. In particular,
+ investor-profile eligibility currently requires a LinkedIn ID.
+
+ ### Compatibility coverage
+
+ Use [Person tests](../../tests/models/test_person.py),
+ [table parsing](../../tests/people/test_markdown_people.py),
+ [dossiers](../../tests/people/test_dossier.py), and [LinkedIn tests](../../tests/linkedin/)
+ (identity, resolver, store, registry and maintenance).
+ [Discovery tests](../../tests/skills/test_persons_in_dataset_skill.py) and
+ [standard profile tests](../../tests/person_profile/test_person_profile_standard.py)
+ cover the roster/profile boundary.
+
+ ## Insights: artifacts and freshness
+
+ Use `lib.insights.InsightFile` for managed insight artifacts.
+ It owns paths, filenames, model suffixes, selection and freshness
+ metadata. Skills must not recreate that logic or edit manifests.
+
+ ### Naming, reading and saving
+
+ Pass the dataset, skill and canonical identifier to `InsightFile`.
+ Preserve existing identifiers and directory conventions. Use its
+ path properties and `content()` rather than reconstructing paths.
+
+ Use the supported `extension` argument for structured artifacts
+ such as JSON; not every insight is Markdown.
+
+ Save through `save()`. Manual precedence is applied during selection;
+ `save()` itself does not prevent overwriting a manual file.
+ Automated generation must preserve manual overrides.
+
+ ### Selection and freshness
+
+ - `find(selection="reusable")`: manual override first, then a ranked
+ model with matching configuration and indexed source revisions.
+ - `find(selection="any")`: preferred existing artifact without a
+ freshness guarantee.
+ - `is_reusable()`: freshness of the exact file/model; does not select
+ alternatives or apply manual-override precedence.
+ - `exists()`: file existence only.
+ - `find_all(selection="any")`: one preferred file per logical insight
+ across startup/community datasets. Bulk reusable selection is unsupported.
+
+ Provide the relevant `source_datasets` and `config_key`, including
+ prompts, settings and supplied dependency content that affect the
+ result. Another insight changing does not automatically invalidate
+ this one.
+
+ Missing indexed source revisions prevent verified reuse. Saving may
+ still write the artifact without new freshness metadata.
+
+ Use `has_insufficient_context()` to recognize the exact
+ `INSUFFICIENT_CONTEXT` sentinel; this is separate from freshness.
+
+ ### Consumers and outputs
+
+ Insight-producing skill APIs return a flat `list[InsightFile]`,
+ including cached results.
+
+ Use `select_insights` for read-only selection of preferred stored
+ artifacts; it does not guarantee freshness.
+
+ Use `dataset_from_insight` to reconcile files in a generated dataset.
+ It can remove obsolete target files, supports `dry_run=True`, and
+ does not itself perform indexing.
+
+ Use shared `strip_model_tag` and `insight_model_slug` helpers when
+ interpreting existing filenames.
+
+ ## Checklist audits
+
+ Use `lib.batch_audit.batch_audit` for structured checklist assessment. Domain
+ questions, statuses, evidence restrictions and synthesis rules belong in the
+ owning skill's configuration.
+
+ `parse_checklist` accepts one H1 title, H2 chapters and H3 checks with self-contained
+ descriptions and optional terminal `**Keywords:**`. Generated numbers are
+ positional. Titles and original IDs remain in audit JSON; descriptions drive
+ per-check prompts and retrieval.
+
+ Explicit audit instructions replace the common instructions. Successful retrieval
+ with no hits still permits assessment from supplied context; missing evidence
+ and technical errors remain distinct. Use shared generation and validation.
+
+ The API returns one canonical JSON `InsightFile` per checklist. Use
+ `validate_audit_document` and `audit_errors` before final output, and
+ `json_to_markdown_table` for common tables. Structural validation does not establish
+ expected checklist coverage or citation accuracy. Synthesis belongs to the caller.
+
+ Audit reuse is per checklist; technical-error artifacts are ineligible.
+ The engine does not synchronize on a cache hit. Callers that require current
+ index state synchronize before lookup and document their additional dependencies.
+
+ ## Ranking workflows
+
+ Use the existing `skills.ranking` engine to compare stored profiles directly.
+ `ranking_top_k` selects ranked IDs; `ranking_rationale` adds explanations while
+ preserving identity and order. `rank_person_rows` supplies structured results;
+ `ranking_persons` supplies the common Markdown table.
+
+ Resolve people through the existing roster and select stored insights through
+ shared readers. Person ranking currently requires LinkedIn IDs; output contact
+ metadata comes from canonical `Person` objects. The engine does not discover
+ people, refresh profiles, copy insights or index documents.
+
+ Keep domain objectives in their owning skill's configuration. Each workflow
+ documents its eligibility, prerequisites, cache dependencies, artifact identity
+ and failure behavior. Ranking orders supplied candidates; it does not enforce
+ domain exclusions. Reuse shared generation, validation and insight lifecycle
+ APIs. See the [ranking skill](../ranking/SKILL.md) for batching, ID recovery and
+ output adapters; keep those details there.
+
+ ## Datasets and storage
+
+ ### Paths and storage
+
+ Use `lib.datasets.paths` and `DatasetLocation` for dataset locations,
+ following `config/storage_domains.json`. Do not hardcode domain roots.
+
+ - `dataset_location` requires an existing dataset.
+ - `find_dataset_location` returns `None` when absent.
+ - `dataset_location_for_domain` resolves a location for creation in
+ an explicit domain; it does not create the dataset.
+
+ Dataset slugs must be unique across domains. Use `list_dataset_names`,
+ `list_all_dataset_names` and `iter_domains` for enumeration.
+
+ Use `get_storage()` for ordinary application-data access, with relative
+ paths and no `..` components. Use domain abstractions such as
+ `InsightFile` for managed artifacts. Use `local_path()` when an
+ integration requires an OS path.
+
+ Application data uses `LOCAL_STORAGE_PATH`. `cache/` and `docling_data/`
+ use the configured machine-local storage root. Docling output is durable
+ parsed data, not disposable cache.
+
+ ### Sources and document references
+
+ Use `list_source_files` and `snapshot_source_files` for ingestible
+ source enumeration and hashing. Preserve their shared exclusion rules.
+
+ Use `parsed_filepath` to map source documents to parsed Markdown.
+
+ `resolve_document_path` returns the closest source-document path and
+ a similarity score. It does not enforce a minimum score or reject
+ ambiguity; callers must apply their acceptance policy.
+
+ Preserve `Chunk` objects and document/page metadata. Use `Chunk.to_md()`
+ for standard evidence rendering and the shared page-marker and
+ spreadsheet helpers when processing those formats.
+
+ ### Ingestion and synchronization
+
+ Use `sync_datasets` to reconcile sources, parsed documents and Qdrant
+ indexing. It can convert documents, call model services, replace
+ indexed content and remove obsolete derived data.
+
+ Conversion, chunking, embeddings, sparse encoding, indexing and
+ manifests belong to the shared pipeline. Do not recreate these stages
+ in skills or modify revision metadata directly. Preserve document
+ replacement ordering: write replacement chunks before deleting
+ obsolete chunks.
+
+ Synchronization returns `IngestionResult` objects with counts and
+ `IngestionFailure` details. Use `raise_on_error=True` when downstream
+ work requires successful ingestion; the default can log failures and
+ return partial results. The retained `force` parameter does not force
+ a rebuild.
+
+ Storage `refresh()` does not fetch external data in the current local
+ implementation. Application-storage mirroring is external to skills.
+ Existing source acquisition through Dealum/website imports is separate;
+ `ensure_startup_dataset` can perform a gated Dealum import and optional sync.
+ It can return a failure status while retaining local data unless strict errors
+ are requested. Document those existing side effects without adding new imports.
+
+ ### Search and evidence
+
+ Use `dataset_search` for semantic retrieval. It synchronizes the dataset
+ before searching, so it is not read-only—even for an empty query.
+
+ Use `raise_on_error=True` when retrieval failures must remain distinct
+ from no matching evidence. Some retrieval failures otherwise return
+ an empty list; ingestion failures still propagate.
+
+ Reranking failures fall back to retrieval order, including when
+ `raise_on_error=True`. Retrieval, reranking and document-diversity
+ policies belong to the shared pipeline.
+
+ ### Dataset state
+
+ Use `is_active_dataset`, `activate_dataset` and `archive_dataset`
+ for refresh-status markers. Archiving changes refresh eligibility;
+ it does not delete the dataset.
+
+ Use the `dataset_maintenance` skill for operational maintenance rather
+ than adding repair or rebuild logic to unrelated skills.
+
+ ## Infrastructure
+
+ ### Configuration and prompts
+
+ Use `load_repository_config(*sections)` for repository configuration
+ and `get_env_var` for environment settings.
+
+ Keep prompts, instructions and skill tuning parameters under `config/`.
+ Required configuration must fail clearly when absent; defaults belong
+ only to explicitly optional settings.
+
+ Use `config_cache_key` for effective configuration. Do not edit
+ compiled configuration caches directly.
+
+ ### Model calls
+
+ Use `generate_markdown` and `generate_json` from
+ `lib.infrastructure.ai_text_generation`. Use the shared embedding
+ and reranking services for retrieval.
+
+ Resolve models and endpoints through `lib.model_config`.
+ Do not hardcode providers, endpoints or credentials in skills.
+
+ Text generation and embeddings must use the shared LiteLLM-backed
+ services. Skills must not call provider APIs directly.
+
+ Use the generation APIs' schema validation and `Review` mechanism;
+ reviewers are supported for both Markdown and JSON. Do not recreate
+ JSON repair or output-correction loops in skills.
+
+ Use `cacheable_prompt_prefix` for reusable shared context where
+ supported. Prompt caching does not replace insight freshness checks.
+
+ ### Provider access and document conversion
+
+ Use `ApifyAdapter`, `DealumAdapter` and `WebSearchAdapter` for their
+ provider operations. Business decisions belong in domain libraries
+ or skills, not generic adapters.
+
+ Use dataset-scoped `QdrantAdapter` operations and preserve dataset
+ filtering within shared collections. Deleting a dataset is different
+ from `QdrantAdmin.delete_collection`, which deletes the whole
+ collection. Adapter initialization can reconcile legacy layouts;
+ do not assume it is read-only.
+
+ Use `document_conversion.convert_document` for individual conversion.
+ It returns `DocumentConversion(markdown, warnings)`. Preserve warnings,
+ normalization and shared format handling. Dataset ingestion owns
+ source selection, persistence and indexing.
+
+ ### Scheduling and retries
+
+ Shared services own their scheduler integration. Skills must not add
+ duplicate scheduler leases around already scheduled services.
+
+ Register new capacity-controlled operations with `register_operation`
+ and a `JobProfile`, then execute through `scheduler.run`.
+ Keep scheduling metadata non-sensitive.
+
+ Preserve shared concurrency, lease and cloud-budget policies.
+ Do not modify scheduler state directly; use `snapshot()` for diagnostics.
+
+ Scheduler waiting and provider request timeouts are separate.
+ Use the shared transient-error retry mechanism and keep provider
+ retries distinct from output-correction attempts. Do not add
+ overlapping retry loops or silently retry permanent errors.
+
+ ### Logging and errors
+
+ Use `get_logger(__name__)` from `lib.infrastructure.logging`.
+ Do not configure separate handlers in skills. Bootstrap modules retain
+ their existing logging initialization to avoid import cycles.
+
+ Log internal progress and failures. Reserve console output for CLI
+ results and user-facing errors.
+
+ At infrastructure boundaries, use `InfrastructureError` with kind,
+ provider, operation and recoverability metadata. Chain the original
+ exception. Domain validation may use standard exceptions.
+
+ Do not silently convert technical failures into missing evidence or
+ successful empty results. Preserve documented fallback behavior,
+ such as optional reranking fallback.
+
+ Core functions raise exceptions; they do not exit the process.
+ CLI wrappers use `lib.cli.run_command` for execution and error handling,
+ and `format_insights` for insight output. The existing Dealum batch CLI
+ implements its own per-startup error aggregation; this is a retained
+ implementation discrepancy, not a pattern for new wrappers.
+
+ ### Known implementation discrepancies
+
+ Some existing adapters still expose standard or provider exceptions;
+ callers cannot assume every failure has InfrastructureError metadata.
+
+ Docling picture-description requests currently use a direct HTTP model
+ endpoint, conflicting with the LiteLLM rule. This requires separate
+ review; it is not authorization to introduce further direct model calls.
+
+ ## Skills and orchestration
+
+ ### Public APIs and structure
+
+ Keep one canonical workflow per skill and minimize public entry points.
+
+ User-facing workflows belong under `skills/`; shared infrastructure
+ and domain primitives belong under `lib/`. Preserve established
+ ownership, including the shared ranking package under `skills/ranking`.
+
+ Skill packages use snake_case names and contain `SKILL.md`.
+ Executable workflows normally have `__init__.py`, a thin `__main__.py`
+ and their implementation. Instruction-only skills need no CLI;
+ operational tools and shared utilities retain their established APIs.
+ Follow the [skill directory structure](references/architecture.md#skill-directory-structure).
+
+ Name the primary workflow API after the skill. Put the dataset argument
+ first where applicable. Preserve existing public parameter names,
+ defaults and return types; do not rename them merely for consistency.
+
+ Insight-producing APIs return `list[InsightFile]`, including cached
+ results. Explicit adapters may return domain objects while sharing
+ the same underlying workflow.
+
+ ### CLI and harness
+
+ Keep Typer entry points in `__main__.py`. They parse and validate
+ arguments, call the public API and format results; business logic
+ belongs in the implementation.
+
+ Use `lib.cli.run_command` and `format_insights` where applicable.
+ Programmatic callers import the public API rather than invoking a CLI.
+
+ Slash-command exposure belongs to `skills.harness`.
+ Document supported commands in the skill's Usage section.
+ Harness registration does not imply bulk-refresh registration.
+
+ ### Bulk-refresh registry
+
+ Register batch workflows in `skills/skill_registry.py` with their
+ callable, supported domains and prerequisite skill keys.
+
+ Bulk callables must support invocation with the dataset as their
+ single positional argument. Other required inputs must not be hidden
+ behind invented defaults just to make a skill batch-compatible.
+
+ Dependencies must reference registered skills and form an acyclic
+ graph. Let bulk_refresh expand dependencies and schedule eligible work;
+ do not reproduce its dependency graph inside individual skills.
+
+ A failed prerequisite causes dependent work to be skipped while
+ unrelated work can continue. Cross-domain dependencies apply to
+ matching prerequisite nodes within the selected dataset scope;
+ they do not automatically select additional datasets.
+
+ ### Composition and compatibility
+
+ A skill may call another skill's public API to compose a workflow.
+ Direct calls do not automatically execute registry dependencies.
+
+ Respect each dependency's contract: some generate or refresh artifacts;
+ others only read existing inputs. Do not turn a read-only dependency
+ into implicit discovery or generation.
+
+ When both registry execution and direct composition invoke a
+ dependency, use compatible settings and reuse its managed artifacts.
+ Do not create competing variants or repeated regeneration.
+
+ Keep prompts and configuration with their owning skill. A new option
+ or dependency must be checked against existing callers, freshness
+ behavior and possible concurrent writes.
+
+ ### Verification
+
+ For new or changed skills, update the relevant harness coverage and
+ contract tests. Verify public return types, CLI argument handling,
+ dependency order, failure propagation and artifact reuse as applicable.
+
+ Preserve explicit distinctions between insight-producing, operational
+ and instruction-only skills; not every skill belongs in bulk refresh.