git:20260603.06a4398 to git:20260731.d2c2e65

- # CLAUDE.md

-

- This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

-

- ## GenAI Engine

- A FastAPI-based service for LLM evaluation, guardrails, and monitoring. Provides evaluations for hallucination, prompt injection, toxicity, PII detection, sensitive data, and more.

- ### Common Commands

- # Install dependencies

- uv sync

-

- # Run development server

- uv run serve

- # or: uvicorn src.server:get_app --reload

-

- # Run tests (unit tests only)

- uv run pytest -m "unit_tests"

-

- # Run tests with coverage (79% minimum required)

- uv run pytest -m "unit_tests" --cov=src --cov-fail-under=79

-

- # Run integration tests

- ./tests/test_remote.sh

-

- # Linting and formatting

- uv run black src/

- uv run isort src/

- uv run mypy src/

-

- # Database migrations

- uv run alembic upgrade head

-

- # Generate changelog

- uv run generate_changelog

-

- # Performance testing (requires separate install)

- uv sync --group performance

- # See locust/README.md for details

- ### Architecture

-

- **Tech Stack**: Python 3.12, FastAPI, SQLAlchemy, PostgreSQL + pgvector, Alembic, Uvicorn

-

- **Core Components**:

-

- - **Routers** (`src/routers/`): API endpoints organized into v1 and v2, separated by feature (tasks, traces, prompts, datasets, rules, evaluators)

- - **Scorer** (`src/scorer/`): Check implementations (hallucination, toxicity, PII, prompt injection, etc.) - each check is a pluggable scorer

- - **Rules Engine** (`src/rules_engine/`): Orchestrates rule evaluation and validation logic

- - **Metrics Engine** (`src/metrics_engine/`): Performance tracking and metrics calculation

- - **Repositories** (`src/repositories/`): Data access layer following repository pattern

- - **DB Models** (`src/db/models/`): SQLAlchemy ORM models

- - **Clients** (`src/clients/`): External service integrations (S3, Azure, Keycloak, LLM providers)

-

- **Key Patterns**:

-

- - **Dependency Injection**: Uses FastAPI's `Depends()` system extensively for database sessions, authentication, and service injection

- - **JWT Authentication**: API key-based authentication with JWT tokens

- - **Repository Pattern**: All database access goes through repository classes, not direct ORM access

- - **Rule-Based Evaluation**: Rules define conditions and actions, executed by the rules engine

- - **OpenTelemetry**: Distributed tracing instrumentation throughout the codebase

- - **Async-First**: Most operations use async/await for I/O operations

-

- **Database**:

-

- - PostgreSQL with pgvector extension for embeddings

- - Alembic for schema migrations

- - Connection pooling via SQLAlchemy

- - Separate read/write connections supported

-

- **Testing**:

-

- - pytest markers: `unit_tests`, `integration_tests`, `aws_live`, `azure_live`

- - Minimum coverage requirement: 79%

- - Integration tests run against deployed environments via `test_remote.sh`

- - Locust for performance testing

- **External Integrations**:

- - OpenAI/Azure OpenAI for LLM-based evaluations

- - S3/Azure Blob Storage for file storage

- - Weaviate for RAG retrievals (optional)

- - Keycloak for OAuth (optional)

- ### Development Notes

- - The API has two versions (v1, v2) with different authentication and feature sets

- - Task-based validation: traces are validated against task rules

- - Agentic workflow support: multi-step agent trace evaluation

- - Pre-commit hooks enforce black, isort, mypy - see CONTRIBUTING.md

- - Database migrations should be reviewed before applying to ensure idempotency

- - Ensure python best coding practices are followed

- - Put all imports at the top of Python files

- - Make sure any unit tests are deterministic. They shouldn't rely on database ordering of results and should clean up any state they create

- ### Multi-tenancy

- When adding endpoints or repository methods that touch task-scoped data, see [`docs/MULTI_TENANCY_DESIGN.md`](docs/MULTI_TENANCY_DESIGN.md) for the four enforcement patterns (path / resource-id / query-param / admin-only), the `TENANT-USER` role rules, and the 404-vs-403 convention. A fuzz test enforces that decorators are applied to new routes.

-

- ### Coding Conventions

-

- - **DB sessions are auto-closed**: Do NOT wrap route handlers in `try/finally: db_session.close()`. The database session obtained via `Depends(get_db_session)` is automatically closed by FastAPI's dependency lifecycle. Adding manual `finally` blocks is unnecessary.

- - **Strongly type all data structures**: API objects, LLM response schemas, request/response bodies, and any data with a known semantic structure must be represented as Pydantic `BaseModel` classes — not raw `dict`, `str`, or `Any`. Typed models provide validation, IDE support, and catch errors at parse time rather than at runtime via `.get()` calls. For LLM calls specifically, pass a Pydantic `BaseModel` class as `response_format` to `client.completion()` instead of `{"type": "json_object"}`; the parsed result is available on `response.structured_output_response`.

-

- #### Strong typing examples

-

- **Good** — nested Pydantic models for structured data:

- ```python

- class SyntheticDataColumn(BaseModel):

- column_name: str

- column_value: str

-

- class SyntheticDataRow(BaseModel):

- id: str

- data: List[SyntheticDataColumn]

-

- class SyntheticDataLLMOutput(BaseModel):

- rows: List[SyntheticDataRow]

- message: str

- ```

-

- **Bad** — JSON string that requires manual parsing:

- ```python

- class SyntheticDataLLMOutput(BaseModel):

- rows_json: str # JSON blob parsed with json.loads, accessed via .get()

- message: str

- ```

-

- **Bad** — untyped dicts for data with known structure:

- ```python

- def process_row(row: Dict[str, Any]) -> None:

- name = row.get("column_name", "") # no validation, typos silently pass

- value = row.get("column_value", "")

- ```