AGENTS.md · diff
git:20260121.e875fbd to git:20260317.886b388
125 added, 29 removed. Audit A to A.
# Repository Guidelines
- ## Project Structure & Module Organization
+ ## Purpose
- - `src/kabigon/`: Python package (typed via `src/kabigon/py.typed`).
- - `src/kabigon/loaders/`: source-specific loaders (YouTube, Twitter/X, Reddit, GitHub, PDF, Playwright, etc.).
- - `src/kabigon/core/`: shared primitives (base `Loader`, exceptions, helpers).
- - `src/kabigon/cli.py`: Typer CLI entrypoint (`kabigon`).
- - `tests/`: pytest suite (see `tests/loaders/` for loader-focused tests).
- - `examples/`: runnable usage samples.
+ This document is the single source of truth for contributor and agent behavior in this repository.
+ ## Project Overview
+
+ Kabigon is a URL content loader library that extracts content from multiple sources and converts results to text or markdown.
+
+ ## Project Structure
+
+ - `src/kabigon/`: typed Python package (`src/kabigon/py.typed`)
+ - `src/kabigon/core/`: shared primitives (`Loader`, exceptions, helpers)
+ - `src/kabigon/loaders/`: source-specific loaders
+ - `src/kabigon/cli.py`: Typer CLI entrypoint (`kabigon`)
+ - `tests/`: pytest suite
+ - `tests/loaders/`: loader-focused tests
+ - `examples/`: runnable usage samples
+
## Build, Test, and Development Commands
- - Setup: `uv sync` (creates/updates `.venv` from `pyproject.toml`/`uv.lock`).
+ - Setup: `uv sync`
- Lint: `uv run ruff check .`
- Format: `uv run ruff format .`
- Type check: `uv run ty check .`
- - Test (with coverage): `uv run pytest -v -s --cov=src tests`
+ - Test (coverage): `uv run pytest -v -s --cov=src tests`
+ - Run one test file: `uv run pytest -v -s tests/loaders/test_youtube.py`
+ - Run one test case: `uv run pytest -v -s tests/loaders/test_youtube.py::test_name`
- Run CLI locally: `uv run kabigon <url>`
- Build wheel: `uv build -f wheel`
- Publish: `uv publish`
- ## Coding Style & Naming Conventions
+ ## Architecture
- - Python 3.12+, `async`-first APIs; prefer small, composable loaders.
- - Formatting/linting: Ruff, 120-char lines (`[tool.ruff]`), sorted imports via Ruff.
- - Naming: modules in `snake_case.py`, classes in `PascalCase`, tests `test_*.py`.
+ ### Core Pattern
- ## Testing Guidelines
+ - The codebase is async-first.
+ - The loader system uses chain of responsibility via `Compose`.
+ - `Compose` tries loaders in order and stops at first success.
- - Framework: `pytest` + `pytest-cov`. Keep tests deterministic and avoid live network calls.
- - Add/modify tests alongside loader changes under `tests/loaders/`.
+ ### Loader Contract
- ## Commit & Pull Request Guidelines
+ - All loaders MUST implement `async def load(url: str) -> str`.
+ - `load_sync(url: str) -> str` is the sync wrapper around async loading.
+ - A loader MUST raise if the URL is unsupported.
+ - A loader SHOULD return an empty string when URL is supported but extraction yields no content.
+ - Exceptions are handled by `Compose` for fallback behavior and logging.
- - Commits typically use concise, imperative subjects (e.g., “Add GitHubLoader”, “Fix test …”).
- - PRs: include a brief description, what URLs/cases were covered, and link issues where applicable; update `README.md`/`examples/` when adding user-facing behavior.
+ ### Default Loader Order
- ## Configuration & Secrets
+ Defined in `src/kabigon/api.py`:
- - Do not commit credentials. Common env vars: `FIRECRAWL_API_KEY`, `FFMPEG_PATH`.
- - For browser-based loaders, install Playwright once: `playwright install chromium`.
+ 1. `PttLoader`
+ 2. `TwitterLoader`
+ 3. `TruthSocialLoader`
+ 4. `RedditLoader`
+ 5. `YoutubeLoader`
+ 6. `ReelLoader`
+ 7. `YoutubeYtdlpLoader`
+ 8. `PDFLoader`
+ 9. `PlaywrightLoader`
- ## Documentation
+ `PlaywrightLoader` is the final generic fallback.
+ ## Coding Conventions
+
+ - Python version: 3.12+
+ - Prefer small, composable loaders.
+ - Naming:
+ - modules: `snake_case.py`
+ - classes: `PascalCase`
+ - tests: `test_*.py`
+ - Type hints MUST use modern Python syntax (for example `list[str]`).
+
+ ## Lint, Format, and Type Rules
+
+ - Ruff line length: 120
+ - Imports are sorted by Ruff/isort rules.
+ - All code changes MUST pass:
+ - `uv run ruff check .`
+ - `uv run ruff format .`
+ - `uv run ty check .`
+
+ ## Testing Rules
+
+ - Framework: `pytest` + `pytest-cov`
+ - Tests MUST be deterministic.
+ - Live network calls in tests MUST be avoided.
+ - Loader changes MUST include related updates in `tests/loaders/`.
+
+ ## Pre-commit Requirement
+
+ If `.pre-commit-config.yaml` exists, all changes MUST pass:
+
+ - `prek run -a`
+
+ ## Dependencies and Environment
+
+ ### Runtime Dependencies
+
+ - `playwright`
+ - `yt-dlp`
+ - `openai-whisper`
+ - `youtube-transcript-api`
+ - `pypdf`
+ - `httpx`
+ - `markdownify`
+ - `firecrawl-py`
+ - `typer`
+ - `rich`
+
+ ### Development Dependencies
+
+ - `pytest`
+ - `pytest-cov`
+ - `ruff`
+ - `ty`
+
+ ### Environment Variables
+
+ - `FFMPEG_PATH`
+ - `FIRECRAWL_API_KEY`
+
+ ## Security and Secrets
+
+ - Credentials MUST NOT be committed.
+ - Outputs from external systems MUST be treated as untrusted until verified.
+
+ ## Commit and Pull Request Guidelines
+
+ - Commit subjects SHOULD be concise and imperative.
+ - PRs SHOULD include:
+ - scope summary
+ - covered URL/source cases
+ - linked issues
+ - User-facing behavior changes MUST update `README.md` and/or `examples/`.
+
+ ## Documentation Rules
+
+ - Documentation MUST be written in Markdown.
+ - Documentation MUST use clear, standard English.
+ - Each document MUST have one well-defined purpose.
+ - Rules MUST be enforceable and unambiguous.
+ - Foundational rules MUST NOT be duplicated across documents.
+ - Each documentation file MUST be under 600 lines.
- Language MUST be concise and precise.
- - Design and structure MUST NOT introduce unnecessary complexity.
+ - Design and structure MUST avoid unnecessary complexity.
- Scope and responsibility boundaries MUST be explicit.
- - Each document MUST have a single, well-defined purpose.
- - Rules MUST be stated in enforceable terms and avoid ambiguity.
- - Foundational rules MUST NOT be duplicated across documents.
- - Each document MUST be less than 600 lines long.
- - Use Markdown for all documentation files.