git:20260325.4637308 to git:20260519.0d0176c

116 added, 78 removed. Audit A to A.

# LiteParse - Agent Documentation

- > This file provides comprehensive context for AI coding agents working on this codebase. Each subdirectory contains its own README with file-specific documentation.

+ > This file provides comprehensive context for AI coding agents working on this codebase.

## Project Overview

- **LiteParse** is an open-source PDF parsing library focused on fast, lightweight document processing with spatial text extraction. It runs entirely locally with zero cloud dependencies by default.

+ **LiteParse** is an open-source PDF parsing library written in **Rust**, focused on fast, lightweight document processing with spatial text extraction. It runs entirely locally with zero cloud dependencies by default.

+ Language bindings are provided for **Node.js/TypeScript** (via napi-rs), **Python** (via PyO3), and **WebAssembly** (via wasm-bindgen).

### Key Capabilities

- **Spatial text extraction** with precise bounding boxes

- - **Flexible OCR** (built-in Tesseract.js or pluggable HTTP servers)

+ - **Flexible OCR** (built-in Tesseract or pluggable HTTP servers)

- **Multi-format support** (PDFs, DOCX, XLSX, PPTX, images via conversion)

- - **TypeScript/Node.js** with both library and CLI interfaces

+ - **Multi-language bindings**: Rust, Node.js/TypeScript, Python, Browser (WASM)

+ - **CLI** available from all installation methods (`cargo`, `npm`, `pip`)

## Directory Structure

```

liteparse/

- ├── src/

- │ ├── core/ # Configuration, types, main orchestrator

- │ ├── engines/ # Pluggable PDF and OCR engines

- │ │ ├── pdf/ # PDF parsing engines (PDF.js, PDFium)

- │ │ └── ocr/ # OCR engines (Tesseract, HTTP)

- │ ├── processing/ # Text extraction and spatial analysis

- │ ├── output/ # Output formatters (JSON, text)

- │ ├── conversion/ # Multi-format conversion to PDF

- │ ├── vendor/ # Bundled dependencies (PDF.js)

- │ ├── index.ts # CLI entry point

- │ └── lib.ts # Library public API

- ├── cli/ # CLI implementation

- ├── ocr/ # Example OCR server implementations

- │ ├── easyocr/ # EasyOCR wrapper server

- │ └── paddleocr/ # PaddleOCR wrapper server

- └── dist/ # Compiled JavaScript output

+ ├── crates/

+ │ ├── liteparse/ # Core Rust library + CLI binary

+ │ │ └── src/

+ │ │ ├── main.rs # CLI entry point (clap)

+ │ │ ├── lib.rs # Library root

+ │ │ ├── parser.rs # LiteParse orchestrator

+ │ │ ├── config.rs # Configuration types and defaults

+ │ │ ├── types.rs # Core data types (ParseResult, TextItem, etc.)

+ │ │ ├── projection.rs # Spatial grid projection (layout reconstruction)

+ │ │ ├── extract.rs # Raw text extraction from PDFium

+ │ │ ├── render.rs # Page rendering / screenshots

+ │ │ ├── conversion.rs # Non-PDF format conversion (LibreOffice, ImageMagick)

+ │ │ ├── ocr_merge.rs # Merging OCR results with native text

+ │ │ ├── error.rs # Error types

+ │ │ ├── ocr/ # OCR engine implementations

+ │ │ │ ├── mod.rs # OcrEngine trait

+ │ │ │ ├── tesseract.rs # Built-in Tesseract OCR

+ │ │ │ └── http_simple.rs # HTTP OCR server client

+ │ │ └── output/ # Output formatters

+ │ │ ├── mod.rs

+ │ │ ├── json.rs

+ │ │ └── text.rs

+ │ ├── liteparse-napi/ # Node.js bindings (napi-rs)

+ │ ├── liteparse-python/ # Python bindings (PyO3 / maturin)

+ │ ├── liteparse-wasm/ # WASM bindings (wasm-bindgen)

+ │ ├── pdfium/ # Rust wrapper around PDFium C API

+ │ └── pdfium-sys/ # PDFium FFI (C → Rust) bindings

+ ├── packages/

+ │ ├── node/ # npm package: TS wrapper + CLI around native binary

+ │ │ └── src/

+ │ │ ├── lib.ts # Public LiteParse class for Node.js

+ │ │ ├── cli.ts # CLI entry point (commander)

+ │ │ └── native.ts # Native binary loader

+ │ ├── python/ # PyPI package: Python wrapper around native binary

+ │ │ └── liteparse/

+ │ │ ├── __init__.py

+ │ │ ├── parser.py # Public LiteParse class for Python

+ │ │ ├── types.py # Python dataclass types

+ │ │ └── cli.py # CLI entry point

+ │ └── wasm/ # WASM npm package

+ ├── ocr/ # Example OCR server implementations

+ │ ├── easyocr/ # EasyOCR wrapper server

+ │ └── paddleocr/ # PaddleOCR wrapper server

+ └── Cargo.toml # Workspace root

```

## Data Flow

- 1. **Input**: File path received (any supported format)

- 2. **Conversion** (if dependencies installed): Non-PDF formats converted to PDF via LibreOffice/ImageMagick

- 3. **PDF Loading**: PDF.js extracts text items, images, metadata

- 4. **OCR** (if enabled): Images rendered and OCR'd for text-sparse areas

+ 1. **Input**: File path or raw bytes received (any supported format)

+ 2. **Conversion** (if needed): Non-PDF formats converted to PDF via LibreOffice/ImageMagick

+ 3. **PDF Loading**: PDFium extracts text items, images, metadata

+ 4. **OCR** (if enabled): Pages rendered and OCR'd for text-sparse areas

5. **Grid Projection**: Spatial reconstruction of text layout using anchor system

6. **Post-processing**: Bounding boxes, text cleanup

7. **Output**: Formatted as JSON or plain text

## Key Design Decisions

- ### 1. Engine Abstraction Pattern

- Both PDF and OCR functionality use interface-based abstraction (`PdfEngine`, `OcrEngine`). This allows:

- - Swapping implementations without changing core logic

- - Auto-detection: HTTP OCR if URL provided, otherwise Tesseract.js

- - Future extensibility for new engines

- - Future possibility of custom conversion engines for non-PDF formats

+ ### 1. Rust Core with Language Bindings

+ The core parsing logic is written in Rust for performance and safety. Language-specific crates expose the same API surface:

+ - `liteparse-napi` → Node.js via napi-rs

+ - `liteparse-python` → Python via PyO3/maturin

+ - `liteparse-wasm` → Browser via wasm-bindgen

- ### 2. Spatial Grid Projection

- The most complex (and important!) part of the codebase (`src/processing/gridProjection.ts`, ~1650 lines). Uses:

+ Each binding crate is thin — it wraps the core `liteparse` crate's types and async API.

+ ### 2. OCR Engine Trait

+ OCR functionality uses a trait-based abstraction (`OcrEngine`). This allows:

+ - Built-in Tesseract (default, compiled in via `tesseract-rs`)

+ - HTTP OCR server client for remote engines

+ - Custom JS-side OCR in the WASM build via a callback interface

+ ### 3. Spatial Grid Projection

+ The most complex (and important!) part of the codebase (`crates/liteparse/src/projection.rs`). Uses:

- **Anchor-based layout**: Tracks text alignment (left, right, center, floating)

- **Forward anchors**: Carry alignment information between lines

- **Column detection**: Identifies multi-column layouts

- **Rotation handling**: Transforms 90°, 180°, 270° rotated text to correct reading order

- **OCR merging**: Combines native PDF text with OCR results, preserving confidence scores and source flags in output

- ### 3. Selective OCR

+ ### 4. Selective OCR

OCR only runs on embedded images where text extraction failed, not the entire document. This balances accuracy with performance.

- ### 4. Configuration Merging

- Uses a default-first approach where users only override what they need. Configuration flows: defaults → file config → CLI options.

+ ### 5. Configuration

+ Uses a default-first approach where users only override what they need. See `crates/liteparse/src/config.rs` for defaults.

- ### 5. Format Conversion via External Tools

- Rather than implementing format parsers, LiteParse converts non-PDF formats using system tools (LibreOffice, ImageMagick) into a single format (PDF). This provides broad format support with minimal code.

+ ### 6. Format Conversion via External Tools

+ Rather than implementing format parsers, LiteParse converts non-PDF formats using system tools (LibreOffice, ImageMagick) into PDF. This provides broad format support with minimal code.

## Common Tasks

### Adding a New Output Format

- 1. Create new file in `src/output/` implementing the formatter

- 2. Add format option to `cli/parse.ts`

- 3. Update `src/core/parser.ts` to use new formatter

+ 1. Create new file in `crates/liteparse/src/output/`

+ 2. Add variant to `OutputFormat` enum in `config.rs`

+ 3. Wire it up in `main.rs` and binding crates

### Adding a New OCR Engine

- 1. Implement `OcrEngine` interface in `src/engines/ocr/`

- 2. Add initialization logic in `src/core/parser.ts`

- 3. Add configuration options in `src/core/types.ts`

+ 1. Implement `OcrEngine` trait in `crates/liteparse/src/ocr/`

+ 2. Add initialization logic in `parser.rs`

+ 3. Add configuration options in `config.rs`

### Modifying Text Extraction Logic

- The processing pipeline is in `src/processing/`. Key files:

- - `gridProjection.ts` - Layout reconstruction (most complex)

- - `bbox.ts` - Bounding box calculation

- - `cleanText.ts` - Text cleanup

+ Key files in `crates/liteparse/src/`:

+ - `projection.rs` — Layout reconstruction (most complex)

+ - `extract.rs` — Raw text item extraction from PDFium

+ - `ocr_merge.rs` — Merging OCR and native text

### Adding CLI Options

- 1. Update `cli/parse.ts` with new Commander.js option

- 2. Add corresponding config field in `src/core/types.ts`

- 3. Update `src/core/config.ts` with default value

- 4. Use the option in `src/core/parser.ts`

+ 1. Add field to `LiteParseConfig` in `config.rs`

+ 2. Add clap arg in `main.rs`

+ 3. Wire through `parser.rs`

+ 4. Expose in binding crates (`liteparse-napi`, `liteparse-python`, `liteparse-wasm`)

- ## Testing Approach

+ ### Adding / Modifying Node.js Wrapper

+ - Edit `packages/node/src/lib.ts` for library API changes

+ - Edit `packages/node/src/cli.ts` for CLI changes

+ - The native binary interface is defined in `packages/node/src/native.ts`

- Currently tested via manual verification with sample documents. The project would benefit from:

- - Unit tests for processing utilities

- - Integration tests with known PDFs

- - Snapshot tests for output formats

+ ### Adding / Modifying Python Wrapper

+ - Edit `packages/python/liteparse/parser.py` for library API changes

+ - Types are in `packages/python/liteparse/types.py`

+ - CLI entry point is `packages/python/liteparse/cli.py`

## Key Dependencies

| Dependency | Purpose |

|------------|---------|

- | `pdfjs-dist` | PDF parsing and text extraction |

- | `@hyzyla/pdfium` | High-quality PDF rendering for screenshots |

- | `tesseract.js` | In-process OCR (zero setup) |

- | `sharp` | Image processing |

- | `commander` | CLI framework |

- | `zod` | Schema validation |

+ | `pdfium` (C library) | PDF text extraction and rendering |

+ | `tesseract-rs` | Built-in OCR engine (optional, via `tesseract` feature) |

+ | `clap` | CLI framework |

+ | `serde` / `serde_json` | Serialization |

+ | `tokio` | Async runtime |

+ | `reqwest` | HTTP client (for OCR server) |

+ | `image` | Image processing (PNG encoding) |

+ | `napi-rs` | Node.js native bindings |

+ | `pyo3` / `maturin` | Python native bindings |

+ | `wasm-bindgen` | WASM bindings |

## Entry Points

- - **CLI**: `src/index.ts` → `cli/parse.ts`

- - **Library**: `src/lib.ts` exports `LiteParse` class and types

- - **Main Class**: `src/core/parser.ts` contains `LiteParse` orchestrator

+ - **Rust CLI**: `crates/liteparse/src/main.rs`

+ - **Rust Library**: `crates/liteparse/src/lib.rs` → `parser.rs` contains `LiteParse` struct

+ - **Node.js**: `packages/node/src/lib.ts` exports `LiteParse` class

+ - **Python**: `packages/python/liteparse/parser.py` exports `LiteParse` class

+ - **WASM**: `crates/liteparse-wasm/` exposes `LiteParse` via wasm-bindgen

## Related Documentation

- These files are key to understanding the codebase and should be referenced for specific implementation details.

- If changes to the codebase are being made, please update the relevant documentation files to reflect those changes and keep them up to date.

- [User-facing documentation](README.md)

- - [src/conversion/README.md](src/conversion/README.md) - Format conversion details

- - [src/core/README.md](src/core/README.md) - Core architecture and configuration

- - [src/engines/README.md](src/engines/README.md) - Engine abstraction and implementations

- - [src/engines/pdf/README.md](src/engines/pdf/README.md) - PDF engines (PDF.js, PDFium)

- - [src/engines/ocr/README.md](src/engines/ocr/README.md) - OCR engines (Tesseract, HTTP)

- - [src/output/README.md](src/output/README.md) - Output formatters

- - [src/processing/README.md](src/processing/README.md) - Text extraction and spatial processing

- - [ocr/README.md](ocr/README.md) - OCR server implementations (EasyOCR, PaddleOCR)

- - [cli/README.md](cli/README.md) - CLI usage and options

+ - [OCR API Specification](OCR_API_SPEC.md)

+ - [WASM package README](packages/wasm/README.md)

+ - [Python package README](packages/python/README.md)

+ - [OCR server examples](ocr/README.md)