CLAUDE.md · git:20260522.e13b0ca · 2026-05-22 · sha256 6c9eaafc777ca12b
CLAUDE.md git:20260522.e13b0caA
Immutable. This exact content is served forever at /api/v1/blob/6c9eaafc777ca12b.
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
Arthur Engine is a Python-based AI/ML monitoring and governance platform with three main components:
- **GenAI Engine**: FastAPI-based REST API for LLM evaluation and guardrailing
- **ML Engine**: Job-based evaluation engine for ML model monitoring
- **Frontend UI**: React + TypeScript + Vite web application
## Technologies
**Backend:**
- Python 3.12 (GenAI Engine), Python 3.13 (ML Engine)
- FastAPI, SQLAlchemy, PostgreSQL with pgVector
- OpenAI/Azure LLMs, LangChain, LiteLLM
- ML Models: Transformers, Sentence Transformers, Spacy
- NER/PII: Presidio, GLiNER
- Alembic for database migrations
**Frontend:**
- React 19, TypeScript, Vite
- **MUI (Material UI) v7** - Primary component library (`@mui/material`, `@mui/icons-material`, `@mui/x-date-pickers`)
- Emotion (`@emotion/react`, `@emotion/styled`) - Styling engine for MUI
- Tailwind CSS - Utility classes for layout supplementing MUI
- TanStack Query/Table, Material React Table
- Zustand for state management
**Infrastructure:**
- Docker, Docker Compose, Helm, AWS ECS
- OpenTelemetry, NewRelic
- Pytest, Coverage, Locust
## Common Commands
### GenAI Engine
```bash
# Setup
cd genai-engine
uv sync --group dev --group linters
# Start PostgreSQL (required)
docker compose up
# Database setup
export POSTGRES_USER=postgres
export POSTGRES_PASSWORD=changeme_pg_password
export POSTGRES_URL=localhost
export POSTGRES_PORT=5432
export POSTGRES_DB=arthur_genai_engine
export GENAI_ENGINE_SECRET_STORE_KEY="some_test_key"
uv run alembic upgrade head
# Run development server
export PYTHONPATH="src:$PYTHONPATH"
uv run serve
# Access at http://localhost:3030/docs
# Testing
uv run pytest -m "unit_tests"
uv run pytest -m "unit_tests" --cov=src --cov-fail-under=79
./tests/test_remote.sh # Integration tests
# Database migrations
uv run alembic revision --autogenerate -m "<message>"
uv run alembic upgrade head
# Code quality
uv run isort src --profile black
uv run autoflake --remove-all-unused-imports --in-place --recursive src
uv run black src
uv run routes_security_check
# Generate API changelog
uv run generate_changelog
```
### ML Engine
```bash
cd ml-engine
# Generate GenAI Engine client
cd scripts
./openapi_client_utils.sh generate python
./openapi_client_utils.sh install python
./install_db_dependencies.sh
cd ..
uv sync
# Run ML Engine
uv run python src/ml_engine/job_agent.py
# Testing
uv sync --group dev
uv run pytest tests/unit
# Code quality
uv run isort src/ml_engine --profile black --check
uv run black --check src/ml_engine
uv run mypy src/ml_engine
```
### Frontend UI
```bash
cd genai-engine/ui
yarn install
yarn dev # Development at localhost:5173
yarn build # Production build
yarn type-check # TypeScript checking
yarn lint # ESLint
yarn format # Prettier (auto-fix)
yarn format:check # Prettier (check only)
yarn generate-api # Generate API client from OpenAPI spec
# Before committing (REQUIRED - CI enforced)
yarn check # Runs type-check, lint, and format:check
```
### Docker Compose (Full Stack)
```bash
cd deployment/docker-compose/genai-engine
cp .env.template .env
# Edit .env with your configuration
docker compose up
# Access at http://localhost:3030/docs
```
## Architecture
### GenAI Engine Structure
```
src/
├── server.py # FastAPI app initialization
├── dependencies.py # Dependency injection (DB, auth, clients)
├── config/ # Configuration management
├── auth/ # Authentication & OAuth (Keycloak, JWT)
├── db_models/ # SQLAlchemy models (19 entity types)
│ ├── task_models.py # Task/use-case definitions
│ ├── rule_models.py # Rule configurations
│ ├── rule_result_models.py # Rule evaluation results
│ ├── inference_models.py # Span/trace data storage
│ └── dataset_models.py # Dataset management
├── repositories/ # Data access layer (24 repositories)
│ ├── tasks_repository.py
│ ├── rules_repository.py
│ ├── inference_repository.py
│ └── span_repository.py # Trace data queries
├── routers/ # API route handlers
│ ├── v1/ # Legacy API endpoints
│ │ ├── trace_api_routes.py
│ │ ├── llm_eval_routes.py
│ │ └── rag_routes.py
│ └── v2/ # Current API version
│ ├── task_management_routes.py
│ ├── rule_management_routes.py
│ ├── validate_routes.py
│ └── feedback_routes.py
├── scorer/ # Evaluation engine
│ ├── scorer.py # Main scorer orchestration
│ ├── llm_client.py # OpenAI/Azure/LiteLLM integration
│ └── checks/ # Evaluation implementations
│ ├── hallucination/ # Claim-based LLM judge
│ ├── prompt_injection/ # DebertaV3 model
│ ├── toxicity/ # RoBERTa classifier
│ ├── pii/ # Presidio + GLiNER
│ ├── sensitive_data/ # Few-shot LLM judge
│ └── regex/ # Pattern-based checks
├── schemas/ # Pydantic request/response models
├── utils/ # Utility modules
│ ├── model_load.py # Download & cache models
│ └── classifiers.py # GPU/device detection
└── validation/ # Input validation logic
```
### ML Engine Structure
```
src/ml_engine/
├── job_agent.py # Main agent polling for jobs
├── job_runner.py # Job execution orchestration
├── job_executor.py # Individual job execution
├── dataset_loader.py # Load data from various sources
├── connectors/ # Data source connectors
│ ├── bigquery/
│ ├── snowflake/
│ ├── postgres/
│ ├── mysql/
│ ├── s3/
│ └── gcs/
├── job_executors/ # Job type handlers
│ ├── backtest_executor.py
│ └── multi_model_eval_executor.py
└── metric_calculators/ # Metric computation
```
### Database Schema (Key Entities)
- **tasks** - Use cases/LLM applications
- **rules** - Evaluation rules configuration
- **rule_results** - Results of rule evaluations
- **spans/inferences** - Trace data (prompts, responses, metadata)
- **datasets** - User data for evaluations
- **feedback** - User feedback on evaluations
- **api_keys** - Authentication credentials
- **secrets** - Encrypted credential storage
- **metrics** - Calculated metrics per task
## Key Evaluation Types
The scorer system in [src/scorer/checks/](src/scorer/checks/) implements:
- **Hallucination Detection**: Claim-based LLM judge technique
- **Prompt Injection**: DebertaV3 model-based detection
- **Toxicity**: RoBERTa toxicity classifier
- **PII Detection**: Presidio + GLiNER for Named Entity Recognition
- **Sensitive Data**: Few-shot LLM judge
- **Regex Checks**: Pattern-based validation
- Custom rules support via extensible plugin system
## Frontend UI Guidelines (MANDATORY)
### Always Use MUI Components
**All frontend UI work MUST use Material UI (MUI) components.** Do NOT use plain HTML elements or custom-styled replacements when an MUI component exists. This applies to every new component, feature, and bugfix.
**Required:** Use MUI components from `@mui/material` for all UI elements:
| Instead of... | Always use... |
| ------------------------ | --------------------------------------------------- |
| `<button>` | `<Button>` from `@mui/material` |
| `<input>`, `<textarea>` | `<TextField>` from `@mui/material` |
| `<select>` | `<Select>` or `<Autocomplete>` from `@mui/material` |
| `<table>` | `<Table>` components or Material React Table |
| `<div>` for layout | `<Box>`, `<Stack>`, `<Paper>`, `<Card>` |
| `<p>`, `<h1>`-`<h6>` | `<Typography>` with appropriate `variant` |
| `<a>` | `<Link>` from `@mui/material` |
| `<ul>/<li>` for menus | `<List>`, `<ListItem>`, `<Menu>`, `<MenuItem>` |
| `<dialog>`, custom modal | `<Dialog>` with `DialogTitle`, `DialogContent`, `DialogActions` |
| Custom alert/banner | `<Alert>` from `@mui/material` |
| Custom tooltip | `<Tooltip>` from `@mui/material` |
| Custom chip/badge | `<Chip>`, `<Badge>` from `@mui/material` |
| Custom icon | Icons from `@mui/icons-material` |
| Custom date picker | Components from `@mui/x-date-pickers` |
### Styling Rules
1. **Use the MUI `sx` prop** as the primary styling method for MUI components. This is the established pattern across the codebase.
2. **Use MUI theme color tokens** — never use raw hex/rgb colors. Use semantic tokens:
- `primary.main`, `primary.light`, `primary.dark`, `primary.50`
- `secondary.main`, `secondary.light`, `secondary.dark`
- `error.main`, `error.50`, `success.main`, `success.light`
- `warning.main`, `info.main`
- `text.primary`, `text.secondary`, `text.disabled`
- `divider`, `action.hover`
3. **Tailwind CSS is only for supplementary layout utilities** (e.g., `min-h-screen`, `flex`, spacing). Never use Tailwind for colors, typography, or component styling that MUI handles.
4. **Use `styled()` from `@mui/material/styles`** only when creating reusable custom-styled components that need to extend MUI components.
### Component Patterns
- **Buttons**: Use `variant="contained"` for primary actions, `variant="outlined"` for secondary, `variant="text"` for tertiary.
- **Typography**: Use semantic variants — `h5`/`h6` for headings, `subtitle1`/`body1`/`body2` for body text, `caption` for helper text.
- **Text Fields**: Use `variant="filled"` as the default text field style.
- **Layout**: Use `<Stack>` for flex layouts, `<Box>` for general containers, `<Card>`/`<Paper>` for elevated surfaces.
- **Icons**: Always source from `@mui/icons-material`. Size with `sx={{ fontSize: N }}` and color with theme tokens.
- **Feedback**: Use `<Alert>` for inline messages, notistack's `enqueueSnackbar` for toast notifications, `<Tooltip>` for hover hints.
### What NOT to Do
- **Do NOT create custom-styled `<div>`, `<span>`, or `<button>` elements** when MUI provides an equivalent component.
- **Do NOT use inline CSS styles** (`style={{ }}`) — use the `sx` prop instead.
- **Do NOT use hardcoded color values** (`#ff0000`, `rgb(...)`) — use MUI theme tokens.
- **Do NOT use Tailwind for colors or typography** — those are handled by MUI's design system.
- **Do NOT introduce new UI libraries** that duplicate MUI functionality.
## Development Workflow
### GenAI Engine Development
```bash
# Initial setup
cd genai-engine
uv sync --group dev --group linters
uv run pre-commit install
# Start PostgreSQL
docker compose up
# Set environment variables (see README.md)
# Run development server
uv run serve
# Before committing
uv run pytest -m "unit_tests"
uv run black src
uv run isort src
# Database schema changes
uv run alembic revision --autogenerate -m "description"
uv run alembic upgrade head
# API changes - generate changelog
uv run generate_changelog
```
### ML Engine Development
```bash
cd ml-engine
# Generate GenAI client
cd scripts
./openapi_client_utils.sh generate python
./openapi_client_utils.sh install python
cd ..
uv sync --group dev --group linters
# Set environment variables
export ARTHUR_API_HOST=https://platform.arthur.ai
export ARTHUR_CLIENT_SECRET=<secret>
export ARTHUR_CLIENT_ID=<id>
# Run
uv run python src/ml_engine/job_agent.py
# Before committing
uv run pytest tests/unit
uv run mypy src/ml_engine
uv run black --check src/ml_engine
```
### Frontend Development
```bash
cd genai-engine/ui
yarn install
yarn dev
# After OpenAPI spec changes
yarn generate-api
# Before committing (REQUIRED - CI enforced)
yarn check # Runs type-check, lint, and format:check
```
## Testing
**GenAI Engine:**
- Unit tests: `uv run pytest -m "unit_tests"`
- Coverage requirement: >= 79%
- Integration tests: `./tests/test_remote.sh`
- Performance tests: Locust-based (see [locust/README.md](genai-engine/locust/README.md))
**ML Engine:**
- Unit tests: `uv run pytest tests/unit`
- Type checking: `uv run mypy src/ml_engine`
**Pre-commit Hooks:**
- Trailing whitespace & end-of-file fixes
- YAML validation
- isort (import sorting)
- autoflake (unused imports removal)
- black (code formatting)
- Routes security validation
- Unit tests execution
## Key Configuration
**Environment Variables (GenAI Engine):**
```bash
# Database
POSTGRES_USER=postgres
POSTGRES_PASSWORD=changeme_pg_password
POSTGRES_URL=localhost
POSTGRES_PORT=5432
POSTGRES_DB=arthur_genai_engine
# GenAI Engine
GENAI_ENGINE_ADMIN_KEY=<admin-key>
GENAI_ENGINE_SECRET_STORE_KEY=<encryption-key>
GENAI_ENGINE_ENVIRONMENT=local|staging|production
GENAI_ENGINE_ENABLE_PERSISTENCE=enabled|disabled
GENAI_ENGINE_OPENAI_PROVIDER=Azure|OpenAI
GENAI_ENGINE_OPENAI_GPT_NAMES_ENDPOINTS_KEYS=<json-config>
# Observability
NEWRELIC_LICENSE_KEY=<key>
OTEL_EXPORTER_OTLP_ENDPOINT=<endpoint>
```
**Environment Variables (ML Engine):**
```bash
ARTHUR_API_HOST=https://platform.arthur.ai
ARTHUR_CLIENT_ID=<client-id>
ARTHUR_CLIENT_SECRET=<client-secret>
GENAI_ENGINE_INTERNAL_API_KEY=<api-key>
```
## Deployment
- **Docker**: Multi-stage builds with CPU and GPU variants
- **Docker Compose**: Full stack deployment in [deployment/docker-compose/genai-engine/](deployment/docker-compose/genai-engine/)
- **Helm**: Kubernetes deployment charts
- **CloudFormation**: AWS ECS deployment templates
- **CI/CD**: GitHub Actions ([.github/workflows/arthur-engine-workflow.yml](.github/workflows/arthur-engine-workflow.yml))
## Key Branches
- `main` - Production releases
- `dev` - Development/staging
- Feature branches created from `dev`
## Important Notes
- GenAI Engine uses Python 3.12, ML Engine uses Python 3.13
- PostgreSQL with pgVector extension required for vector similarity
- Pre-commit hooks enforce code quality and run tests
- API changes require changelog generation via `uv run generate_changelog`
- Model files are downloaded and cached on first use
- GPU support optional but improves performance for model-based checks
- **Frontend: Always use MUI components** — never use plain HTML elements when MUI provides an equivalent. See "Frontend UI Guidelines" section above for full details.
## Skill routing
When the user's request matches an available skill, invoke it via the Skill tool. When in doubt, invoke the skill.
Key routing rules:
- Product ideas/brainstorming → invoke /office-hours
- Strategy/scope → invoke /plan-ceo-review
- Architecture → invoke /plan-eng-review
- Design system/plan review → invoke /design-consultation or /plan-design-review
- Full review pipeline → invoke /autoplan
- Bugs/errors → invoke /investigate
- QA/testing site behavior → invoke /qa or /qa-only
- Code review/diff check → invoke /review
- Visual polish → invoke /design-review
- Ship/deploy/PR → invoke /ship or /land-and-deploy
- Save progress → invoke /context-save
- Resume context → invoke /context-restore