CLAUDE.md@genai-engine git:20260603.06a4398A

Immutable. This exact content is served forever at /api/v1/blob/81be94068df2b008.

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## GenAI Engine

A FastAPI-based service for LLM evaluation, guardrails, and monitoring. Provides evaluations for hallucination, prompt injection, toxicity, PII detection, sensitive data, and more.

### Common Commands

```bash
# Install dependencies
uv sync

# Run development server
uv run serve
# or: uvicorn src.server:get_app --reload

# Run tests (unit tests only)
uv run pytest -m "unit_tests"

# Run tests with coverage (79% minimum required)
uv run pytest -m "unit_tests" --cov=src --cov-fail-under=79

# Run integration tests
./tests/test_remote.sh

# Linting and formatting
uv run black src/
uv run isort src/
uv run mypy src/

# Database migrations
uv run alembic upgrade head
uv run alembic revision --autogenerate -m "description"

# Generate changelog
uv run generate_changelog

# Performance testing (requires separate install)
uv sync --group performance
# See locust/README.md for details
```

### Architecture

**Tech Stack**: Python 3.12, FastAPI, SQLAlchemy, PostgreSQL + pgvector, Alembic, Uvicorn

**Core Components**:

- **Routers** (`src/routers/`): API endpoints organized into v1 and v2, separated by feature (tasks, traces, prompts, datasets, rules, evaluators)
- **Scorer** (`src/scorer/`): Check implementations (hallucination, toxicity, PII, prompt injection, etc.) - each check is a pluggable scorer
- **Rules Engine** (`src/rules_engine/`): Orchestrates rule evaluation and validation logic
- **Metrics Engine** (`src/metrics_engine/`): Performance tracking and metrics calculation
- **Repositories** (`src/repositories/`): Data access layer following repository pattern
- **DB Models** (`src/db/models/`): SQLAlchemy ORM models
- **Clients** (`src/clients/`): External service integrations (S3, Azure, Keycloak, LLM providers)

**Key Patterns**:

- **Dependency Injection**: Uses FastAPI's `Depends()` system extensively for database sessions, authentication, and service injection
- **JWT Authentication**: API key-based authentication with JWT tokens
- **Repository Pattern**: All database access goes through repository classes, not direct ORM access
- **Rule-Based Evaluation**: Rules define conditions and actions, executed by the rules engine
- **OpenTelemetry**: Distributed tracing instrumentation throughout the codebase
- **Async-First**: Most operations use async/await for I/O operations

**Database**:

- PostgreSQL with pgvector extension for embeddings
- Alembic for schema migrations
- Connection pooling via SQLAlchemy
- Separate read/write connections supported

**Testing**:

- pytest markers: `unit_tests`, `integration_tests`, `aws_live`, `azure_live`
- Minimum coverage requirement: 79%
- Integration tests run against deployed environments via `test_remote.sh`
- Locust for performance testing

**External Integrations**:

- OpenAI/Azure OpenAI for LLM-based evaluations
- S3/Azure Blob Storage for file storage
- Weaviate for RAG retrievals (optional)
- Keycloak for OAuth (optional)

### Development Notes

- The API has two versions (v1, v2) with different authentication and feature sets
- Task-based validation: traces are validated against task rules
- Agentic workflow support: multi-step agent trace evaluation
- Pre-commit hooks enforce black, isort, mypy - see CONTRIBUTING.md
- Database migrations should be reviewed before applying to ensure idempotency
- Ensure python best coding practices are followed
- Put all imports at the top of Python files
- Make sure any unit tests are deterministic. They shouldn't rely on database ordering of results and should clean up any state they create

### Multi-tenancy

The engine supports multi-tenant organizations. Tasks belong to exactly one org; tenant API keys are scoped to a single org via `api_keys.org_id` and reach all tasks within it. Admin keys (`org_id = NULL`) keep cross-org access. Existing tasks migrate into the `default` org; system tasks (`is_system_task=True`) migrate into the `system` org.

Tenant provisioning is gated by `GENAI_ENGINE_DEMO_MODE` (default off). When enabled, `POST /api/v2/tenant/signup` creates `(org, task, api_key)` in one transaction and returns the raw key once. When disabled, the endpoint returns 404. Keep this flag **off** for customer production deployments.

When adding endpoints or repository methods that touch task-scoped data, see [`docs/MULTI_TENANCY_DESIGN.md`](docs/MULTI_TENANCY_DESIGN.md) for the four enforcement patterns (path / resource-id / query-param / admin-only), the `TENANT-USER` role rules, and the 404-vs-403 convention. A fuzz test enforces that decorators are applied to new routes.

### Coding Conventions

- **DB sessions are auto-closed**: Do NOT wrap route handlers in `try/finally: db_session.close()`. The database session obtained via `Depends(get_db_session)` is automatically closed by FastAPI's dependency lifecycle. Adding manual `finally` blocks is unnecessary.
- **Strongly type all data structures**: API objects, LLM response schemas, request/response bodies, and any data with a known semantic structure must be represented as Pydantic `BaseModel` classes — not raw `dict`, `str`, or `Any`. Typed models provide validation, IDE support, and catch errors at parse time rather than at runtime via `.get()` calls. For LLM calls specifically, pass a Pydantic `BaseModel` class as `response_format` to `client.completion()` instead of `{"type": "json_object"}`; the parsed result is available on `response.structured_output_response`.

#### Strong typing examples

**Good** — nested Pydantic models for structured data:
```python
class SyntheticDataColumn(BaseModel):
    column_name: str
    column_value: str

class SyntheticDataRow(BaseModel):
    id: str
    data: List[SyntheticDataColumn]

class SyntheticDataLLMOutput(BaseModel):
    rows: List[SyntheticDataRow]
    message: str
```

**Bad** — JSON string that requires manual parsing:
```python
class SyntheticDataLLMOutput(BaseModel):
    rows_json: str  # JSON blob parsed with json.loads, accessed via .get()
    message: str
```

**Bad** — untyped dicts for data with known structure:
```python
def process_row(row: Dict[str, Any]) -> None:
    name = row.get("column_name", "")  # no validation, typos silently pass
    value = row.get("column_value", "")
```