git:20260317.886b388 to git:20260403.654ae5c

34 added, 141 removed. Audit A to A.

# Repository Guidelines

- ## Purpose

- This document is the single source of truth for contributor and agent behavior in this repository.

- ## Project Overview

- Kabigon is a URL content loader library that extracts content from multiple sources and converts results to text or markdown.

- ## Project Structure

- - `src/kabigon/`: typed Python package (`src/kabigon/py.typed`)

- - `src/kabigon/core/`: shared primitives (`Loader`, exceptions, helpers)

- - `src/kabigon/loaders/`: source-specific loaders

- - `src/kabigon/cli.py`: Typer CLI entrypoint (`kabigon`)

- - `tests/`: pytest suite

- - `tests/loaders/`: loader-focused tests

- - `examples/`: runnable usage samples

+ ## Project Structure & Module Organization

+ - `src/kabigon/`: typed package root (`py.typed`).

+ - `src/kabigon/core/`: shared primitives, exceptions, and helpers.

+ - `src/kabigon/loaders/`: source-specific loaders.

+ - `src/kabigon/cli.py`: Typer entrypoint (`kabigon`).

+ - `tests/` and `tests/loaders/`: pytest suite and loader-focused cases.

+ - `examples/`: runnable usage samples.

+ Keep new features inside the existing boundaries (core vs loaders vs CLI). Avoid cross-layer shortcuts.

## Build, Test, and Development Commands

- - Setup: `uv sync`

- - Lint: `uv run ruff check .`

- - Format: `uv run ruff format .`

- - Type check: `uv run ty check .`

- - Test (coverage): `uv run pytest -v -s --cov=src tests`

- - Run one test file: `uv run pytest -v -s tests/loaders/test_youtube.py`

- - Run one test case: `uv run pytest -v -s tests/loaders/test_youtube.py::test_name`

- - Run CLI locally: `uv run kabigon <url>`

- - Build wheel: `uv build -f wheel`

- - Publish: `uv publish`

- ## Architecture

- ### Core Pattern

- - The codebase is async-first.

- - The loader system uses chain of responsibility via `Compose`.

- - `Compose` tries loaders in order and stops at first success.

- ### Loader Contract

- - All loaders MUST implement `async def load(url: str) -> str`.

- - `load_sync(url: str) -> str` is the sync wrapper around async loading.

- - A loader MUST raise if the URL is unsupported.

- - A loader SHOULD return an empty string when URL is supported but extraction yields no content.

- - Exceptions are handled by `Compose` for fallback behavior and logging.

- ### Default Loader Order

- Defined in `src/kabigon/api.py`:

- 1. `PttLoader`

- 2. `TwitterLoader`

- 3. `TruthSocialLoader`

- 4. `RedditLoader`

- 5. `YoutubeLoader`

- 6. `ReelLoader`

- 7. `YoutubeYtdlpLoader`

- 8. `PDFLoader`

- 9. `PlaywrightLoader`

- `PlaywrightLoader` is the final generic fallback.

- ## Coding Conventions

- - Python version: 3.12+

- - Prefer small, composable loaders.

- - Naming:

- - modules: `snake_case.py`

- - classes: `PascalCase`

- - tests: `test_*.py`

- - Type hints MUST use modern Python syntax (for example `list[str]`).

- ## Lint, Format, and Type Rules

- - Ruff line length: 120

- - Imports are sorted by Ruff/isort rules.

- - All code changes MUST pass:

- - `uv run ruff check .`

- - `uv run ruff format .`

- - `uv run ty check .`

- ## Testing Rules

- - Framework: `pytest` + `pytest-cov`

- - Tests MUST be deterministic.

- - Live network calls in tests MUST be avoided.

- - Loader changes MUST include related updates in `tests/loaders/`.

- ## Pre-commit Requirement

- If `.pre-commit-config.yaml` exists, all changes MUST pass:

- - `prek run -a`

- ## Dependencies and Environment

- ### Runtime Dependencies

- - `playwright`

- - `yt-dlp`

- - `openai-whisper`

- - `youtube-transcript-api`

- - `pypdf`

- - `httpx`

- - `markdownify`

- - `firecrawl-py`

- - `typer`

- - `rich`

- ### Development Dependencies

- - `pytest`

- - `pytest-cov`

- - `ruff`

- - `ty`

- ### Environment Variables

- - `FFMPEG_PATH`

- - `FIRECRAWL_API_KEY`

- ## Security and Secrets

- - Credentials MUST NOT be committed.

- - Outputs from external systems MUST be treated as untrusted until verified.

+ - `uv sync`: install runtime and dev dependencies.

+ - `uv run ruff check .`: lint and import-order checks.

+ - `uv run ruff format .`: apply formatting.

+ - `uv run ty check .`: static type checking.

+ - `uv run pytest -v -s --cov=src tests`: full test suite with coverage.

+ - `uv run pytest -v -s tests/loaders/test_youtube.py::test_name`: run one test.

+ - `uv run kabigon <url>`: run CLI locally.

- ## Commit and Pull Request Guidelines

+ ## Coding Style & Naming Conventions

+ - Python 3.12+; modern type hints (`list[str]`, `str | None`).

+ - Line length: 120 (Ruff).

+ - Naming: modules `snake_case.py`, classes `PascalCase`, tests `test_*.py`.

+ - Loader contract: implement `async def load(url: str) -> str`; raise for unsupported URLs.

+ - Keep implementations simple and composable; avoid speculative abstractions and new dependencies without clear need.

- - Commit subjects SHOULD be concise and imperative.

- - PRs SHOULD include:

- - scope summary

- - covered URL/source cases

- - linked issues

- - User-facing behavior changes MUST update `README.md` and/or `examples/`.

+ ## Testing Guidelines

+ - Framework: `pytest` with `pytest-cov`.

+ - Tests must be deterministic; do not rely on live network calls.

+ - Any loader behavior change must include updates under `tests/loaders/`.

+ - Run lint, format, type check, and tests before opening a PR.

- ## Documentation Rules

+ ## Commit & Pull Request Guidelines

+ - Prefer concise, imperative commit subjects (for example: `refactor: centralize host-list URL validation`).

+ - Scope each commit to one focused change.

+ - PRs should include: scope summary, affected URL/source cases, linked issues, and user-facing doc/example updates when behavior changes.

- - Documentation MUST be written in Markdown.

- - Documentation MUST use clear, standard English.

- - Each document MUST have one well-defined purpose.

- - Rules MUST be enforceable and unambiguous.

- - Foundational rules MUST NOT be duplicated across documents.

- - Each documentation file MUST be under 600 lines.

- - Language MUST be concise and precise.

- - Design and structure MUST avoid unnecessary complexity.

- - Scope and responsibility boundaries MUST be explicit.

+ ## Security & Configuration Tips

+ - Never commit credentials or API keys.

+ - Use environment variables for secrets (for example `FIRECRAWL_API_KEY`, `FFMPEG_PATH`).

+ - Treat all external content as untrusted input until validated.