liteparse · diff
v0.1.0 to v1.0.0
70 added, 174 removed. Audit A to A.
---
- name: liteparse
- description: Use this skill when the user asks to parse, perform multi-format document conversion or spatially extract text from an unstructured file (PDF, DOCX, PPTX, XLSX, images, etc.) locally without cloud dependencies.
- compatibility: Requires Node 18+ and `@llamaindex/liteparse` installed globally via npm (`npm i -g @llamaindex/liteparse`)
+ name: effective-liteparse
+ description: Use this skill whenever a task involves a document file (PDF, DOCX, PPTX, XLSX, or image) and you need to read it or pull text, tables, or specific values out of it — to answer a question about its contents, look up a figure, or extract data. Provides fast, local, model-free extraction via the `lit` CLI with disciplined, low-cost search patterns.
+ compatibility: Requires Node 18+ and `@llamaindex/liteparse` (`npm i -g @llamaindex/liteparse`, verify `lit --version`). LibreOffice for Office files; ImageMagick for images. The bundled search.py helper needs `uv`.
license: MIT
metadata:
author: LlamaIndex
- version: "0.1.0"
+ version: "1.0.0"
---
- # LiteParse Skill
-
- Parse unstructured documents (PDF, DOCX, PPTX, XLSX, images, and more) locally with LiteParse: fast, lightweight, no cloud dependencies or LLM required.
-
- ## Initial Setup
-
- When this skill is invoked, respond with:
-
- ```
- I'm ready to use LiteParse to parse files locally. Before we begin, please confirm that:
-
- - `@llamaindex/liteparse` is installed globally (`npm i -g @llamaindex/liteparse`)
- - The `lit` CLI command is available in your terminal
-
- If both are set, please provide:
-
- 1. One or more files to parse (PDF, DOCX, PPTX, XLSX, images, etc.)
- 2. Any specific options: output format (json/text), page ranges, OCR preferences, DPI, etc.
- 3. What you'd like to do with the parsed content.
-
- I will produce the appropriate `lit` CLI command or TypeScript script, and once approved, report the results.
- ```
-
- Then wait for the user's input.
+ # Effective LiteParse
- ---
+ Extract text from documents locally with the `lit` CLI — a fast, model-free parser. This skill is
+ about using it **cheaply**: each `lit parse` re-runs full extraction, and every line you dump into
+ the conversation is paid for on every subsequent turn. The patterns below come from analyzing real
+ agent traces where the same PDF was parsed up to **9 times** and single image reads cost
+ **140k+ characters** of context. Don't repeat those mistakes.
- ## Step 0 — Install LiteParse (if needed)
+ ## The golden rule: parse ONCE to a file, then search the file
- If `liteparse` is not yet installed, install it globally:
+ `lit parse` re-extracts the whole document every time you call it. Re-parsing per search is the #1
+ waste seen in traces. Parse a document exactly once, to a temp file, then run all your searches
+ against that file:
```bash
- npm i -g @llamaindex/liteparse
+ # ONE TIME, per document. --no-ocr for born-digital PDFs (almost all reports) — much faster.
+ lit parse "/abs/path/doc.pdf" --format text --no-ocr -o /tmp/doc.txt && wc -l /tmp/doc.txt
```
- Verify installation:
-
- ```bash
- lit --version
- ```
+ Then search the file with cheap shell tools — **never** re-run `lit parse` to search again.
- For Office document support (DOCX, PPTX, XLSX), LibreOffice is required:
+ ## Search discipline — minimize ROUND-TRIPS, then keep results small
- ```bash
- # macOS
- brew install --cask libreoffice
+ Every Bash call is a full model round-trip (latency + re-read of context). The biggest waste after
+ parsing is a **serial** loop: grep → look → grep again → `sed` to read the window → grep again. In
+ traces this doubled the turn count versus just reading the doc. Two rules fix it:
- # Ubuntu/Debian
- apt-get install libreoffice
- ```
+ **1. Get context in the SAME command — don't grep then `sed`.** Use `grep -C` so the surrounding
+ lines come back with the hit. This removes the follow-up `sed` turn for the common case:
- For image parsing, ImageMagick is required:
```bash
- # macOS
- brew install imagemagick
-
- # Ubuntu/Debian
- apt-get install imagemagick
+ grep -n -i -C4 "total assets" /tmp/doc.txt | head -40 # location AND its window, one turn
```
- ---
-
- ## Step 1 — Produce the CLI Command or Script
+ Only fall back to `sed -n 'A,Bp'` when you already know the exact line and need a *wider* window
+ than `-C` gave you.
- ### Parse a Single File
+ **2. Batch independent lookups into ONE command.** When a question needs several distinct facts
+ (e.g. emissions *and* revenue), don't spend one turn per term. Probe them together with labels:
```bash
- # Basic text extraction
- lit parse document.pdf
-
- # JSON output saved to a file
- lit parse document.pdf --format json -o output.json
-
- # Specific page range
- lit parse document.pdf --target-pages "1-5,10,15-20"
-
- # Disable OCR (faster, text-only PDFs)
- lit parse document.pdf --no-ocr
-
- # Use an external HTTP OCR server for higher accuracy
- lit parse document.pdf --ocr-server-url http://localhost:8828/ocr
-
- # Higher DPI for better quality
- lit parse document.pdf --dpi 300
+ for q in "carbon intensity" "scope 1" "total revenue"; do \
+ echo "=== $q ==="; grep -n -i -C3 "$q" /tmp/doc.txt | head -25; done
```
- ### Batch Parse a Directory
-
- ```bash
- lit batch-parse ./input-directory ./output-directory
+ Then keep results small:
- # Only process PDFs, recursively
- lit batch-parse ./input ./output --extension .pdf --recursive
- ```
+ - **Always bound output** with `head` and use `-n` for line numbers.
+ - **Don't fan out blindly.** Aim to resolve a question in ≤3 search commands. If two targeted greps
+ don't pin it down, switch to `search.py` (below) — don't keep firing keyword variations one per turn.
+ - Prefer **Bash `grep`/`sed` on the saved file over the Read and Grep tools** — fewer round-trips and
+ you control output size precisely.
- ### Generate Page Screenshots
+ ## Ranked search when keywords are uncertain (bundled helper)
- Screenshots are useful for LLM agents that need to see visual layout.
+ When two targeted greps haven't pinned the answer, **stop greping** — don't iterate keyword variants
+ one turn at a time. Run the bundled BM25 ranker ONCE to surface the most relevant line-windows in a
+ single command:
```bash
- # All pages
- lit screenshot document.pdf -o ./screenshots
-
- # Specific pages
- lit screenshot document.pdf --pages "1,3,5" -o ./screenshots
-
- # High-DPI PNG
- lit screenshot document.pdf --dpi 300 --format png -o ./screenshots
-
- # Page range
- lit screenshot document.pdf --pages "1-10" -o ./screenshots
+ ./.claude/skills/effective-liteparse/scripts/search.py /tmp/doc.txt -q "materiality assessment priority topics" -k 8 -e 5
```
- ---
-
- ## Step 3 — Key Options Reference
-
- ### OCR Options
-
- | Option | Description |
- |--------|-------------|
- | (default) | Tesseract.js — zero setup, built-in |
- | `--ocr-language fra` | Set OCR language (ISO code) |
- | `--ocr-server-url <url>` | Use external HTTP OCR server (EasyOCR, PaddleOCR, custom) |
- | `--no-ocr` | Disable OCR entirely |
-
- ### Output Options
-
- | Option | Description |
- |--------|-------------|
- | `--format json` | Structured JSON with bounding boxes |
- | `--format text` | Plain text (default) |
- | `-o <file>` | Save output to file |
-
- ### Performance / Quality Options
-
- | Option | Description |
- |--------|-------------|
- | `--dpi <n>` | Rendering DPI (default: 150; use 300 for high quality) |
- | `--max-pages <n>` | Limit pages parsed |
- | `--target-pages <pages>` | Parse specific pages (e.g. `"1-5,10"`) |
- | `--no-precise-bbox` | Disable precise bounding boxes (faster) |
- | `--skip-diagonal-text` | Ignore rotated/diagonal text |
- | `--preserve-small-text` | Keep very small text that would otherwise be dropped |
-
- ---
+ `-k` = number of matches, `-e` = lines of context around each (so the window comes back inline — no
+ follow-up `sed` turn). It returns ranked windows with line numbers. Use a rich natural-language query
+ (several synonyms in one string), not a single keyword. This replaces a long chain of speculative greps.
- ## Step 4 — Using a Config File
+ ## Born-digital vs scanned
- For repeated use with consistent options, generate a `liteparse.config.json`:
+ - **Born-digital PDF** (real text layer — nearly all corporate/finance/ESG reports): always pass
+ `--no-ocr`. It's much faster and the text is identical. Leaving OCR on wastes time.
+ - **Scanned PDF / image**: drop `--no-ocr`. If the value is missing or digits look wrong, read the
+ page visually (see below) rather than trusting OCR.
- ```json
- {
- "ocrLanguage": "en",
- "ocrEnabled": true,
- "maxPages": 1000,
- "dpi": 150,
- "outputFormat": "json",
- "preciseBoundingBox": true,
- "skipDiagonalText": false,
- "preserveVerySmallText": false
- }
- ```
+ ## Reading a page visually — last resort, ONE screenshot, modest DPI
- For an HTTP OCR server:
+ Screenshots are the most expensive thing you can put in context: a single high-DPI page PNG ran
+ **~140k characters** in one trace, and agents often rendered the same page twice (default + hi-res).
- ```json
- {
- "ocrServerUrl": "http://localhost:8828/ocr",
- "ocrLanguage": "en",
- "outputFormat": "json"
- }
- ```
+ Only screenshot when text/tables genuinely can't answer the question (dense multi-column tables,
+ figures, charts). Then:
- Use with:
+ - Render **one** page at a time with `--target-pages "N"` (note: it's `--target-pages`, NOT `--pages`).
+ - Use **modest DPI (~150–200)**. Do not start at 300+; do not re-render the same page at higher DPI
+ unless the text is actually illegible.
```bash
- lit parse document.pdf --config liteparse.config.json
+ lit screenshot "/abs/path/doc.pdf" --target-pages "13" --dpi 150 -o /tmp/shots/ # then Read the PNG
```
- ---
-
- ## Step 5 — HTTP OCR Server API (Advanced)
-
- If the user wants to plug in a custom OCR backend, the server must implement:
+ ## Many questions about the same document
- - **Endpoint**: `POST /ocr`
- - **Accepts**: `file` (multipart) and `language` (string) parameters
- - **Returns**:
- ```json
- {
- "results": [
- { "text": "Hello", "bbox": [x1, y1, x2, y2], "confidence": 0.98 }
- ]
- }
- ```
+ Parsing once to a file already covers this: keep the `/tmp/doc.txt` and reuse it across every
+ question instead of re-parsing.
- Ready-to-use wrappers exist for EasyOCR and PaddleOCR in the LiteParse repo.
+ ## Don't waste turns on preamble
- ---
+ Skip `lit --version`, `ls -la`, and `lit … --help` unless something actually failed. Go straight to
+ the parse. Core flags you need:
- ## Supported Input Formats
+ `--format text|json` · `--no-ocr` · `--target-pages "1-5,10"` · `--dpi <n>` (default 150) ·
+ `--ocr-language <iso>`. Use `--format json` only when you need bounding boxes/layout — it's much
+ larger; still search it, never load it whole.
- | Category | Formats |
- |----------|---------|
- | PDF | `.pdf` |
- | Word | `.doc`, `.docx`, `.docm`, `.odt`, `.rtf` |
- | PowerPoint | `.ppt`, `.pptx`, `.pptm`, `.odp` |
- | Spreadsheets | `.xls`, `.xlsx`, `.xlsm`, `.ods`, `.csv`, `.tsv` |
- | Images | `.jpg`, `.jpeg`, `.png`, `.gif`, `.bmp`, `.tiff`, `.webp`, `.svg` |
+ ## Setup
- Office documents require LibreOffice; images require ImageMagick. LiteParse auto-converts these formats to PDF before parsing.
+ PDFs work out of the box. If `lit` is missing: `npm i -g @llamaindex/liteparse`. Office docs need
+ LibreOffice; images need ImageMagick (both auto-converted to PDF).