AGENTS.md · diff
git:20260411.939420d to git:20260412.fdbffbf
34 added, 4 removed. Audit A to A.
# ACE-Step Studio — Agent Guidelines
## Project Overview
ACE-Step Studio is a portable, self-contained AI music generation application. It bundles Python (with CUDA/PyTorch), Node.js, and the ACE-Step 1.5 ML pipeline into a single-terminal launcher. No installation required — extract and run.
**GitHub:** https://github.com/timoncool/ACE-Step-Studio
**Upstream:** Forked from fspecii/ace-step-ui, heavily modified.
## Architecture
```
ACE-Step-Studio/
├── run.bat # Single-terminal launcher (prod)
├── run-dev.bat # 3-terminal dev mode with Vite HMR
├── install.bat # First-time setup
├── python/ # Portable Python 3.11 + CUDA 12.8
├── node/ # Portable Node.js
├── ffmpeg/ # Portable FFmpeg
├── models/ # HuggingFace model cache
├── ACE-Step-1.5/ # ML pipeline (Gradio + FastAPI on port 8001)
│ ├── acestep/ # Python package
│ │ ├── acestep_v15_pipeline.py # Main entry point
│ │ ├── llm_inference.py # 5Hz LM (vLLM/nanovllm)
│ │ ├── api/http/ # FastAPI routes (/v1/init, /v1/models, /health)
│ │ └── ui/gradio/api/ # Gradio API routes
│ └── checkpoints/ # DiT + LM model weights
└── app/ # Web application
├── index.html # Entry point (uses CDN for Tailwind + esm.sh)
├── App.tsx # Root component, generation logic
├── components/ # React components
│ ├── CreatePanel.tsx # Generation UI (Simple + Custom modes)
│ ├── SongList.tsx # Song list with model badges + generation time
│ ├── RightSidebar.tsx # Song details panel
│ ├── Sidebar.tsx # Left sidebar (GPU stats, navigation)
│ └── ...
├── services/api.ts # API client + Song data transformation
├── context/I18nContext.tsx # i18n provider
├── i18n/ # Translations (en, ru, zh, ja, ko)
├── types.ts # TypeScript interfaces
├── dist/ # Vite production build (served by Express)
└── server/ # Express backend (port 3001)
├── src/index.ts # Server entry + static serving + pipeline manager
├── src/config/ # Config (ports, paths, pipeline settings)
├── src/routes/ # API routes (songs, generate, pipeline, etc.)
├── src/services/
│ ├── pipeline-manager.ts # Spawns/monitors/restarts Python process
│ ├── acestep.ts # Gradio client, job queue, generation logic
│ ├── gradio-client.ts # Gradio connection management
│ └── storage/ # Local file storage
└── src/db/ # SQLite migrations + connection pool
```
## Three-Process Architecture (Single Terminal)
Express is the supervisor. One `run.bat` → one terminal window:
1. **Express** (port 3001) — API server + serves Vite build as static files
2. **Python/Gradio** (port 8001) — ML pipeline, spawned as child process by Express
3. **Vite** — NOT running in prod. Only in dev mode (`run-dev.bat`)
Pipeline Manager (`pipeline-manager.ts`):
- Spawns Python with `child_process.spawn()`, pipes stdout/stderr with `[Gradio]` prefix
- Parses stdout for readiness: `"Running on local URL"` → state becomes `ready`
- Health check: polls `GET /gradio_api/info` every 10s, 3 consecutive failures → restart
- Auto-restart with exponential backoff (500ms → 15s, max 10 attempts)
- Graceful shutdown: `taskkill /pid PID /T /F` on Windows
- Opens browser automatically when pipeline is ready (first start only)
- Auto-finds free port if 3001 is busy (tries up to +10)
## Model Switching
Models switch **in-process** via Gradio's `POST /v1/init` API. NO process restart.
- Express `/api/generate/switch-model` → calls `POST http://localhost:8001/v1/init`
- Python `initialize_models_for_request()` handles unload + reload + gc + cache cleanup
- `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` set in run.bat to reduce VRAM fragmentation
- Frontend polls `/api/generate/model-status` for state transitions: `unloading → loading → ready`
**NEVER spawn a new Gradio process to switch models.** Always use `/v1/init`.
## Database
SQLite via `better-sqlite3`. DB file: `app/server/data/acestep.db` (or `app/data/acestep.db`).
Key tables: `songs`, `users`, `playlists`, `playlist_songs`, `liked_songs`, `followers`, `generation_jobs`.
Songs table notable columns:
- `dit_model`, `lm_model`, `lm_backend` — which models generated the track
- `generation_time` — seconds from job start to completion (INTEGER)
- `generation_params` — full params JSON
When adding new song fields: update ALL places:
1. `migrate.ts` — ALTER TABLE
2. ALL SELECT queries in `songs.ts` (there are 6+) and `index.ts` (search)
3. `acestep.ts` — `job.result` object
4. `generate.ts` — INSERT statements (there are 2: success + fallback)
5. `api.ts` — `Song` interface + BOTH `transformSongs` mappers
6. `App.tsx` — BOTH song mapping functions (`refreshSongsList` + public songs)
7. `types.ts` — `Song` interface
**This is the #1 source of bugs.** Data gets lost because of manual mapping in App.tsx.
## i18n
5 languages: `en`, `ru`, `zh`, `ja`, `ko`. Files in `app/i18n/`.
- All user-visible strings MUST use `t('key')` from `useI18n()`
- Every component using `t()` MUST have `const { t } = useI18n()` and `import { useI18n } from '../context/I18nContext'`
- Universal terms (Facebook, GPU, VRAM) stay hardcoded — no point translating
- Server-side stage names (`starting`, `generating`) are i18n keys, translated on frontend
## Frontend Build
- `npx vite build` → outputs to `app/dist/`
- Express serves `dist/` as static files in production
- CDN dependencies: Tailwind CSS (`cdn.tailwindcss.com`), React/Lucide via esm.sh importmap
- **Lucide icons from CDN may not bundle properly.** Use inline SVG for icons not in the import list
- Helmet CSP allows: `'self'`, `'unsafe-inline'`, `'unsafe-eval'`, `cdn.tailwindcss.com`, `esm.sh`
- After ANY frontend change, run `vite build` to update `dist/`
## Key Rules
1. **Never spawn new Gradio processes** — use `/v1/init` API for model switching
2. **Always rebuild dist/** after frontend changes — prod serves static files
3. **Check ALL mappers** when adding song fields — App.tsx has manual mapping that drops unknown fields
4. **Every `t()` needs `useI18n()`** — missing import crashes the entire app
5. **Test in browser** — check console for errors after changes
6. **Default model:** `marcorez8/acestep-v15-xl-turbo-bf16` (7.5GB, BF16 precision)
7. **Default LM:** `acestep-5Hz-lm-0.6B` (smallest, fastest)
8. **Portable app** — no internet dependency at runtime (CDN is a known tech debt)
## Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `MANAGE_PIPELINE` | `false` | Set `true` for Express to spawn Python |
| `PYTHON_PATH` | `./python/python.exe` | Path to Python executable |
| `ACESTEP_PATH` | `./ACE-Step-1.5` | Path to ML pipeline directory |
| `DEFAULT_MODEL` | `marcorez8/acestep-v15-xl-turbo-bf16` | DiT model to load on startup |
| `PORT` | `3001` | Express server port |
| `ACESTEP_PORT` | `8001` | Gradio pipeline port |
| `PYTORCH_CUDA_ALLOC_CONF` | `expandable_segments:True` | VRAM fragmentation mitigation |
## Development
```bash
# Dev mode (3 terminals, Vite HMR on port 3000)
run-dev.bat
# Production mode (1 terminal, static files on port 3001)
run.bat
# Rebuild frontend after changes
cd app && npx vite build
```
+ ## Video Studio
+
+ Video generator lives in `app/components/VideoGeneratorModal.tsx` (~3000 lines).
+
+ **Rendering pipeline** (both preview and export):
+ 1. Background (image/video with cover-fit + zoom)
+ 2. Preset visualizer (NCS Circle, Spectrum, etc.) — position/scale from config
+ 3. Text layers (title, artist) — draggable on canvas
+ 4. Lyrics overlay (LRC-synced) — 3 styles: lines, scroll, karaoke
+ 5. Post-processing effects (shake, glitch, VHS, etc.)
+
+ **Export flow:**
+ 1. Browser renders frames on canvas (same code as preview)
+ 2. Frames sent to server in chunks of 50 (base64 JPEG)
+ 3. Server: `/api/render-video/start` → `/frames` → `/finish`
+ 4. Server runs local `ffmpeg.exe` (NVENC if available)
+ 5. Returns MP4 binary
+
+ **LRC (timestamped lyrics):**
+ - Generated by ACE-Step via DiT cross-attention alignment (`auto_lrc=true`)
+ - Stored in `songs.lrc_content` (TEXT column)
+ - Parsed by `app/services/lrc-parser.ts`
+ - Gradio output position: `data[36]` as `{value: "...", visible: true}`
+ - Section markers (`[Verse]`, `[Chorus]`, etc.) = any `[text in brackets]`
+
+ **Lucide icons:** removed from CDN importmap, bundled from node_modules by Vite.
+
## Known Tech Debt
- - [ ] CDN dependencies (Tailwind, esm.sh) should be bundled locally for true offline
- - [ ] App.tsx manual song mapping should use spread or a shared transformer
- - [ ] TrainingPanel has hardcoded English strings (Section titles, placeholders)
+ - [ ] CDN dependencies (Tailwind CSS, esm.sh for React/GenAI) should be bundled locally
+ - [ ] App.tsx has 3 manual song mappers — every new field must be added to all 3
+ - [ ] Video export: current approach (JPEG chunks over HTTP) should use node-canvas + ffmpeg stdin pipe
+ - [ ] LRC timestamps from ACE-Step can be inaccurate (cross-attention alignment is approximate)
+ - [ ] TrainingPanel has hardcoded English strings
- [ ] Progress bar for generation (predict() is blocking in Gradio)
- [ ] Stop generation button not implemented
- - [ ] Hot LM model switching not implemented in `init_service_wrapper`
+ - [ ] WYSIWYG: no resize handles (only scroll-to-resize)
+ - [ ] Video Studio FX tab labels (VHS Tape, Hue Shift desc fields) not i18n'd