Immutable. This exact content is served forever at /api/v1/blob/5c46c3928650872a.
---
name: standards_and_architecture
description: The definitive source of truth for all coding conventions, environment constraints, and architecture layouts in the SICTIC-AI repository. The AI must review this skill before writing, modifying, or refactoring any Python code to ensure strict adherence.
---
# Standards and Architecture
This skill acts as a mandatory pre-flight checklist. Review these standards before writing or modifying any Python code.
## Data Storage Layout (`LOCAL_STORAGE_PATH`)
The AI and all skills must strictly adhere to the configured storage-domain
layout. The local root is defined by `LOCAL_STORAGE_PATH`. Google Drive access
is isolated in the standalone `gdrive_sync` utility; general application storage is always
local. The domain roots are defined in
`config/storage_domains.json`; code must use `lib.datasets.paths` rather than
hardcoding storage paths.
* `config/` — A folder structure filled with `.md` files containing prompts and settings. The `config_load` skill compiles these into a single JSON dictionary at runtime.
* `storage/startups/<dataset_name>/datasets/` — raw startup data rooms.
* `__active_dataset__.md` — A readable marker file. If present, it signals to batch jobs (like `bulk_refresh`) that this startup should be actively processed.
* `storage/startups/<dataset_name>/insights/` — generated startup AI reports and profiles.
* `storage/community/<dataset_name>/datasets/` — primary community/member datasets.
* `storage/community/<dataset_name>/insights/` — generated community AI reports and profiles.
* `storage/generated/<dataset_name>/datasets/` — searchable datasets assembled from insights, such as member profiles used by ranking.
* `storage/generated/<dataset_name>/insights/` — insights associated with generated datasets, when applicable.
* `docling_data/datasets2md/<domain>/<dataset_name>/datasets/` — durable machine-local Docling-extracted Markdown storage. This is generated, but it is not disposable cache and is not synchronized to Google Drive.
* `storage/<domain>/<dataset_name>/insights/persons-in-dataset-<dataset_name>-manual.md` — manually maintained person lists generated by `persons_in_dataset`. Once created, these files are treated as user-edited source-of-truth inputs and are never overwritten.
* `gdrive_sync_state/<pairing-id>/` — durable Google Drive synchronization baseline and changes token. Never remove this during cache cleanup.
* `cache/` — disposable runtime cache and temporary operational state.
*(Note on Data Sharing: If a Deal Lead needs access to a specific startup's data, do not break this structure. Instead, use Google Drive "Shortcuts" to create a custom, safely-named viewing folder for the human, while preserving this configured hierarchy for the AI.)*
## Coding Standards
* **Parameter Order:** If `dataset` (or `dataset_name`) is a parameter for a routine, function, or skill API, it must **always** be the first parameter.
* **Refactoring Protocol:**
* Before refactoring any code, you must first ask clarifying questions to the user.
* After getting answers, propose the exact refactoring approach/plan.
* **Only** after the user explicitly agrees to the proposed approach are you allowed to execute the refactor and edit the code.
* **Testing Protocol:**
* If you write or use temporary Python scripts to test or verify functionality in the codebase, you must always ask the user afterwards if that script should be converted into a formal `pytest` unit/integration test.
* **Environment:** All code is executed via the `sictic-env` Conda environment, bootstrapped from `environment.yml` by `install.sh`.
* **Python Path:** The installer writes the repository root to a `.pth` file in the Conda environment, so `import skills.<SKILL_NAME>.<SCRIPT_NAME>` and `import lib.<MODULE>` resolve from any CWD without setting `PYTHONPATH`.
* **Imports:** User-facing skill code lives under `skills.<SKILL_NAME>.<SCRIPT_NAME>`. Shared infrastructure lives under `lib.<MODULE>` (logger, env, adapters, slugify, etc.).
* **Naming Conventions:**
* All skill directory names must only contain underscores (`_`). **Never** hyphens (`-`).
* All Python files (`.py`) must be strictly lowercase with underscores (snake_case). No uppercase letters (e.g., `config_load.py`, not `Config_Load.py`).
* Dataset names are always strictly `lowercase`.
* **Data Structures (Person Wrapper):** Whenever passing person data through LinkedIn resolution and downstream skills, use the standard `lib.people.model.Person` wrapper:
```python
Person(
full_name=str, # Sanitized: Latin characters preserved, emojis/symbols stripped
linkedin_id=str, # Canonical slug (e.g., 'john-doe'). This is the unique ID and cache filename
email_addresses=list, # Normalized lowercase email addresses for matching and contact tables
linkedin_profile=dict, # Full JSON payload returned by the scraper
dossier=list, # High-value complete Chunk documents (e.g., full resumes/CVs text)
mentions=list, # Isolated Chunk mentions where the person appears downstream
person_profile_markdown=str, # Generated Markdown profile when available
)
```
* **Output Standards:**
* All output files generated by skills must be saved under the configured `insights_root` for the dataset's storage domain.
* All output filenames must end with the slugified name of the model used (e.g., `-<MODEL_NAME>.md`). Note: `<MODEL_NAME>` strictly refers to the model part without the provider prefix, normalized by `lib.slugify` (e.g., `ollama/qwen3.5:4b` becomes `qwen3.5-4b`).
* **File Naming Convention:** All generated user-facing output files (like Markdown reports) must strictly follow `kebab-case`. The script must build a raw filename string and pass it through the global `slugify()` utility (found in `lib.slugify`) to guarantee uniform sanitization of spaces, underscores, and accents before appending the `.md` extension. System identifiers (like dictionary keys, dataset IDs, and `.py` module names) should remain in `snake_case`.
* **Logging vs. Printing:**
* Extensive logging must be implemented using the centralized logger utility.
* Every script must import it at the top level: `from lib.logger import get_logger` followed by `logger = get_logger(__name__)`.
* This ensures all logs, across all skills and levels (INFO, WARNING, ERROR, DEBUG), flow into the single `{{REPO_ROOT}}/logs/sictic-ai.log` file chronologically.
* `print()` and `rich` consoles are reserved **strictly** for final output delivery to the user/GUI. All internal state, progress steps, and warnings must use the `logger`.
* **CLI/Routing:**
* `typer` is used for function descriptions, CLI routing, and argument parsing.
* Every executable skill must have a clearly defined Typer entry point.
* **Thin CLI (Separation of Concerns):** Typer entry points (`__main__.py`) must contain **zero** business logic. They only parse arguments, handle top-level exception catching, and pass execution to core engine modules.
* **Configuration & Prompts:** Hardcoding prompts inside Python files is forbidden. All prompts, instructions, and tuning parameters must be stored externally and fetched dynamically via the `config_load()` utility from the Google Drive `config` folder. Never use default values when sourcing variables from configurations (e.g., avoid `.get(key, default)`); always use direct key access (e.g., `config['key']`) so the script fails cleanly and loudly via `KeyError` if the configuration is missing.
* **Encapsulated Dependencies:** Functions should instantiate their own single-use dependencies (like Docling/Rclone for ingestion) rather than forcing parent functions to instantiate and pass them down, unless injecting a shared persistent state like a database connection pool.
* **Error Handling & Boundaries:**
* Strict error reporting is mandatory.
* Utility modules, adapters, and core functions must **never** call `sys.exit()` or `raise typer.Exit()`. They must raise standard Python Exceptions (e.g., `ValueError`, `RuntimeError`).
* Only the top-level CLI wrapper (`@app.command()`) is allowed to catch these exceptions, log them, and execute a graceful `raise typer.Exit(code=1)`.
* **LLM Inference (LiteLLM):** All LLM inferences (both Text Generation and Embeddings) MUST be routed through the `litellm` library. Direct API calls to specific providers (like Ollama, OpenAI, Anthropic) are strictly forbidden for inference.
## Architecture
Two top-level Python packages: `skills/` (user-facing CLI skills, each with `SKILL.md` + `__main__.py`) and `lib/` (shared infrastructure, no `SKILL.md`).
### Architectural Layers
`lib/` is no longer a flat utility folder. It is organized around storage,
dataset ingestion, insight lifecycle, and business domains. Skills should
compose these shared libraries rather than duplicating path, cache, identity, or
freshness logic.
1. **Runtime and Storage Foundation**
* `lib.env`: Environment access and `.env` loading.
* `lib.logger`: Centralized logging to `logs/sictic-ai.log`.
* `lib.model_config`: Runtime model and endpoint configuration.
* `lib.services_gateway`: IPC gateway for concurrency control across LLM,
embedding, and Docling calls.
* `lib.slugify`: Canonical filename and identifier slugification.
* `lib.storage`: Relative-path storage abstraction. Application data is
routed to `LOCAL_STORAGE_PATH`; machine-local runtime data under `cache/`
and `docling_data/` is routed to `LOCAL_DATA_PATH`/`REPO_PATH`.
* `lib.adapters`: External integration adapters such as Apify, Dealum,
Docling, Qdrant, and web search. Adapters should stay provider-specific and
must not own business-domain decisions.
2. **Dataset Domain (`lib.datasets`)**
* Owns dataset discovery, storage-domain path resolution, active/archive
markers, source conversion, chunking, embedding, Qdrant indexing, and
semantic search.
* `paths.py` is the only supported API for dataset locations. It reads
`config/storage_domains.json` and exposes `DatasetLocation`,
`dataset_location(...)`, `dataset_location_for_domain(...)`,
`dataset_raw_path(...)`, `dataset_parsed_path(...)`,
`dataset_insights_path(...)`, and dataset listing helpers.
* `source.py` discovers ingestible files and ignores non-text/binary/source
control files.
* `conversion.py` reconciles raw source files into durable parsed Markdown
using Docling.
* `indexing.py` chunks parsed Markdown, embeds chunks through LiteLLM, and
reconciles complete document replacements in Qdrant.
* `manifest.py` stores conversion/indexing freshness state and the indexed
dataset revision used by insight freshness checks.
* `ingestion.py` orchestrates conversion and indexing through
`sync_datasets(...)`.
* `search.py` exposes `dataset_search(...)`; callers must treat it as a
side-effecting operation because it synchronizes the dataset before
querying Qdrant.
3. **Insight Domain (`lib.insights`)**
* Owns generated insight paths, filenames, model suffixes, freshness
manifests, model fallback selection, manual overrides, and generated
dataset hydration.
* Use `lib.insights.InsightFile` for generated Markdown reports. Do not
hardcode insight paths and do not use legacy `insight_filepath` or
`insight_refresh` patterns.
* `InsightFile.find(selection="reusable")` selects a fresh insight by checking manual
overrides first, then ranked models whose prompt hash and source dataset
revisions match the current state.
* `InsightFile.find(selection="any")` selects the best available existing insight when
strict freshness is not required.
* `InsightFile.find_all(skill=..., datasets=..., selection="any")` discovers
stored logical insights and returns one selected `InsightFile` for each.
Bulk reusable selection is not supported because the expected prompt and
source datasets are not available during discovery.
* `hydration.py` builds generated datasets from selected insight files, for
example `storage/generated/sictic-members-investor-profile/datasets/`.
4. **Startup Domain (`lib.startups`)**
* Owns startup identity, standard startup dossier layout, active startup
dataset creation, and external startup source hydration.
* `identity.py` resolves canonical startup slugs and configured aliases.
* `dossier.py` creates the standard startup raw and parsed subdirectories
(`data-room`, `linkedin`, `dealum`, `snippets`, `post-deal`) and active
markers.
* `sources.py` exposes `ensure_startup_dataset(...)`, which verifies local
startup data and optionally refreshes it from Dealum.
* `startups/dealum/` owns Dealum reconciliation, exact name/application-code
matching, application Markdown rendering, manifest hashing, document
downloads, and import results.
5. **People Domain (`lib.people`)**
* Owns the canonical `Person` model, person identity matching/merging,
editable person discovery lists, and person dossier assembly.
* `model.py` defines the `Person` dataclass and email/name matching helpers.
* `discovery.py` owns `persons_in_dataset(...)`. It first reads the manual
`persons_in_dataset` insight if present; otherwise it discovers cached,
web, and in-dataset LinkedIn identities, writes a manual editable insight,
and never overwrites that manual source of truth.
* `dossier.py` builds a person's document dossier and incidental mentions
from parsed dataset content and semantic search.
6. **LinkedIn Domain (`lib.linkedin`)**
* Owns LinkedIn identifier parsing, profile payload cleanup, dataset-local
LinkedIn cache files, the unresolved-profile registry, and profile
resolution/scraping orchestration.
* `LinkedInResolver` is dataset-scoped. It reads/writes profile JSON under
the dataset raw path's `linkedin/` directory and updates the shared
missing-profile registry in `cache/linkedin_missing_profiles.json`.
* `lib.linkedin` intentionally depends on `lib.people.model.Person`, and
`lib.people.discovery` intentionally calls `LinkedInResolver`. Treat this
as the current people/LinkedIn boundary; do not introduce additional
identity models.
7. **Small Utilities**
* `lib.insights.dataset_from_insight(target_dataset, source_datasets,
skill)` reconciles a caller-named generated dataset and returns the
selected `list[InsightFile]`.
* `lib.ephemeral_dataset` prepares temporary generated datasets in Qdrant.
* `lib.json_parser`, `lib.litellm_cleanup`, and `lib.runtime_noise` are small
cross-cutting helpers.
### `skills/` — user-facing CLI skills
Skills are user-facing orchestration packages. A skill may call other skills
when it is composing user-facing workflows, but reusable infrastructure belongs
in `lib/`.
* `config_load`: Compiles Markdown configuration into runtime prompt/settings
dictionaries.
* `dataset_chat`: User-facing RAG question answering over datasets. Shared
ingestion/search code belongs in `lib.datasets`.
* `dataset_maintenance`: Operational diagnostics, migration, pruning, and
generated-dataset hydration commands.
* `gdrive_sync`: The only package that talks directly to Google Drive for
application-storage synchronization.
* `linkedin_maintenance`: Human-in-the-loop tooling for missing profiles,
manual imports, and registry diagnosis.
* `person_profile`, `investor_profile`, `startup_profile`, `team_profile`,
`startup_traction`, `dd_checks`: Insight-producing profile and due-diligence
skills. They must save reports through `InsightFile` and their primary Python
APIs must return a flat `list[InsightFile]`, including cached results.
* `ranking`: Shared ranking skill package used by matchmaker-style skills.
Its `ranking_top_k.py`, `ranking_rationale.py`, and `ranking_persons.py`
live under `skills/ranking`, not `lib/`.
* `expert_search`, `potential_investors`, `advocates`, `suggested_startups`:
Matching workflows that compose generated investor profiles, startup
profiles, dataset search, ranking, and per-investor insight outputs. Their
primary Python APIs also return a flat `list[InsightFile]`.
* `bulk_refresh`: Batch orchestration across active datasets and skills.
* `dealum_import`: CLI wrapper around `lib.startups.dealum`.
## Skill Directory Structure
Every skill must strictly adhere to the following Python package structure. Custom subdirectories like `scripts/` are forbidden.
```text
{{REPO_ROOT}}/
├── lib/ # GLOBAL: Shared infrastructure (env, logger, adapters, …)
│
├── skills/
│ └── <SKILL_NAME>/ # The individual skill package
│ ├── SKILL.md # Mandatory: AI instructions and metadata for the skill
│ ├── __init__.py # Makes the skill directory a Python package
│ ├── __main__.py # Thin CLI Entry Point (Typer app)
│ ├── <skill_name>.py # Primary Class/API facade - Logic Entry Point
│ ├── core/ # (Optional) Subroutines and heavy business logic
│ │ └── __init__.py
│ └── utils/ # (Optional) Skill-specific stateless helpers
│ └── __init__.py
│
└── tests/ # GLOBAL: pytest suite
```
* **`SKILL.md`:** This file is mandatory and must always be present in the root of the skill directory. It contains the prompt instructions and metadata that OpenClaw, Nemoclaw, Claude, and other harnesses use to understand and trigger the skill. For user-facing skills exposed by the harness, the `## Usage` section should provide a universal, copy-pastable slash command formatted as: `conda run -n sictic-env python -m skills.harness /<COMMAND> [args]`.
* **`__main__.py`:** Contains zero business logic. Handles Typer CLI routing, argument parsing, and top-level exception catching. Allows execution via `python -m skills.<SKILL_NAME>`. This is the *only* allowed Typer CLI entry point for a skill.
* **`<skill_name>.py`:** The primary programmatic entry point (orchestrator/facade) for the skill.
* **Function Naming Rule:** The main API function inside this file must be named identically to the skill itself (e.g., `def startup_profile(...):` inside `startup_profile.py`). External modules must call this specific function.
* **Parameter Standardization:** When applicable, the primary API function of a skill must always take `dataset_name: str` as its very first parameter. This ensures consistency for automated batch runners like `bulk_refresh`.
* **`core/` & `utils/` (Optional):** These directories should only be generated if the skill is complex enough to require them.
* **`core/`:** Contains internal subroutines and heavy lifting.
* **Local `utils/`:** Contains stateless helper functions strictly unique to this specific skill. (For cross-skill shared infrastructure, use the top-level `lib/` directory instead.)