git:20260516.97ec6c2 to git:20260612.8db3bb9
1 added, 1 removed. Audit A to A.
# LLM Review Module
## Overview
The `infrastructure/llm/review/` directory generates manuscript reviews using local Ollama LLM models. The public API is a set of standalone functions — there is no `ReviewGenerator` class, no `ReviewConfig` dataclass, and no `ReviewIO` class. `generator.py` is the public-API façade; the focused submodules (`generation.py`, `quality.py`, `ollama_setup.py`, `saving.py`, `review_analysis.py`, `metrics.py`) implement the details.
## Directory Structure
```mermaid
flowchart LR
- R[/infrastructure/llm/review//]
+ R[infrastructure/llm/review/]
R --> META[AGENTS.md · __init__.py]
R --> GEN[generator.py<br/>Public API façade + _REVIEW_REGISTRY]
R --> GENR[generation.py<br/>Core streaming, retry, text extraction]
R --> FMT[formatting.py<br/>Markdown report formatting]
R --> IO[io.py<br/>I/O façade — re-exports review_analysis + saving]
R --> MET[metrics.py<br/>ReviewMetrics · SessionMetrics · StreamingMetrics]
R --> OLL[ollama_setup.py<br/>Client creation + model selection]
R --> PIP[pipeline_runner.py<br/>End-to-end pipeline orchestration]
R --> QUAL[quality.py<br/>validate_review_quality · ReviewType]
R --> ANA[review_analysis.py<br/>quality_summary · compliance · action_items]
R --> SAV[saving.py<br/>save_review_outputs · save_single_review]
classDef d fill:#0f172a,stroke:#0f172a,color:#fff
classDef code fill:#1e3a8a,stroke:#0f172a,color:#fff
classDef doc fill:#0f766e,stroke:#0f172a,color:#fff
class R d
class GEN,GENR,FMT,IO,MET,OLL,PIP,QUAL,ANA,SAV code
class META doc
```
## Key Components
### Review Generation Façade (`generator.py`)
`generator.py` owns `_REVIEW_REGISTRY`, the mapping of review types to template classes and temperatures. The four named public entry points are generated from the registry by `_make_review_fn`:
```python
from infrastructure.llm.review.generator import (
generate_llm_executive_summary, # ManuscriptExecutiveSummary, temp 0.3
generate_quality_review, # ManuscriptQualityReview, temp 0.3
generate_methodology_review, # ManuscriptMethodologyReview, temp 0.3
generate_improvement_suggestions, # ManuscriptImprovementSuggestions, temp 0.4
)
# All four share this signature:
# (client: LLMClient, text: str, model_name: str = "", temperature: float = <default>)
# -> tuple[str | None, ReviewMetrics]
review_text, metrics = generate_llm_executive_summary(
client=client,
text=manuscript_text,
model_name="gemma3:4b",
)
```
The `_REVIEW_REGISTRY` dict — add new review types here to generate additional entry points without duplicating logic:
```python
# _REVIEW_REGISTRY: dict[str, tuple[review_name, template_class, default_temperature]]
# Defined in generator.py; not part of the public API but documents the pattern.
```
`generator.py` also re-exports from its submodules for backwards compatibility (all names flow through `__init__.py`):
```python
from infrastructure.llm.review.generator import (
validate_review_quality,
get_manuscript_review_system_prompt,
create_review_client,
select_and_start_ollama_model,
warmup_model,
extract_manuscript_text,
generate_review_with_metrics,
generate_translation,
)
```
### Core Generation Logic (`generation.py`)
`generation.py` implements PDF text extraction, streaming with heartbeat monitoring, retry loops, and repetition deduplication.
#### Text Extraction
```python
from infrastructure.llm.review.generation import extract_manuscript_text
from pathlib import Path
# Returns (str | None, ManuscriptInputMetrics)
# - None if the PDF does not exist (graceful skip)
# - Raises PDFValidationError if the file exists but is unreadable
text, input_metrics = extract_manuscript_text(
pdf_path=Path("output/project/pdf/combined.pdf"),
max_input_length=500_000, # optional; pass client.config.max_input_length
)
```
#### Low-Level Review Generation
```python
from infrastructure.llm.review.generation import generate_review_with_metrics
from infrastructure.llm.templates import ManuscriptQualityReview
# Returns (str | None, ReviewMetrics)
# - None if the LLM returned empty or all retries produced off-topic content
review_text, metrics = generate_review_with_metrics(
client=client,
text=manuscript_text,
review_type="quality_review",
review_name="quality review",
template_class=ManuscriptQualityReview,
model_name="gemma3:4b",
temperature=0.3,
max_tokens=None, # defaults to client.config.long_max_tokens
max_retries=1,
)
```
#### Translation
```python
from infrastructure.llm.review.generation import generate_translation
from infrastructure.llm.templates import TRANSLATION_LANGUAGES
# Returns (str | None, ReviewMetrics); None on failure (non-fatal)
translated, metrics = generate_translation(
client=client,
text=abstract_text,
language_code="zh",
model_name="gemma3:4b",
)
```
### Ollama Setup (`ollama_setup.py`)
```python
from infrastructure.llm.review.ollama_setup import (
get_manuscript_review_system_prompt,
create_review_client,
select_and_start_ollama_model,
warmup_model,
)
# Select model and ensure Ollama is running
model_name = select_and_start_ollama_model()
# Create an LLMClient configured for manuscript review
client = create_review_client(model_name)
# Warm up the model to measure initial throughput
warmup_tokens_per_sec = warmup_model(client)
```
### Review I/O (`io.py`, `saving.py`, `review_analysis.py`)
`io.py` is the preferred import path — it re-exports from `saving.py` and `review_analysis.py`:
```python
from infrastructure.llm.review.io import save_review_outputs, save_single_review
from infrastructure.llm.review.io import generate_review_summary
from infrastructure.llm.review.io import calculate_quality_summary
from infrastructure.llm.review.io import calculate_format_compliance_summary
from infrastructure.llm.review.io import extract_action_items
from infrastructure.llm.review.io import SessionMetrics
```
**Saving reviews:**
```python
from infrastructure.llm.review.io import save_review_outputs, save_single_review
# Save all reviews with session metadata
save_review_outputs(
reviews={"executive_summary": text1, "quality_review": text2},
output_dir=Path("output/project/llm"),
model_name="gemma3:4b",
pdf_path=Path("output/project/pdf/combined.pdf"),
session_metrics=session_metrics, # SessionMetrics
)
# Save a single review file
save_single_review(
name="executive_summary",
content=review_text,
output_dir=Path("output/project/llm"),
)
```
**Analysis and summaries:**
```python
from infrastructure.llm.review.io import (
generate_review_summary,
calculate_quality_summary,
calculate_format_compliance_summary,
extract_action_items,
)
summary = generate_review_summary(reviews, session_metrics)
quality = calculate_quality_summary(reviews)
compliance = calculate_format_compliance_summary(reviews)
actions = extract_action_items(reviews)
```
### Review Metrics (`metrics.py`)
Four dataclasses track generation statistics:
```python
from infrastructure.llm.review.metrics import ReviewMetrics
from infrastructure.llm.review.metrics import ManuscriptInputMetrics
from infrastructure.llm.review.metrics import SessionMetrics
from infrastructure.llm.review.metrics import StreamingMetrics
from infrastructure.llm.review.metrics import estimate_tokens
```
- `ReviewMetrics` — single review: input/output chars, words, tokens, time, preview
- `ManuscriptInputMetrics` — PDF extraction: chars, words, tokens, truncated flag
- `SessionMetrics` — full session: manuscript + reviews dict + timing + model info
- `StreamingMetrics` — streaming: chunk counts, bytes/sec, first-chunk time
- `estimate_tokens(text)` — returns `int`
### Quality Validation (`quality.py`)
```python
from infrastructure.llm.review.quality import (
validate_review_quality,
ReviewType, # Literal alias for the five review type strings
ReviewQualityDetails, # TypedDict — shape of the details return value
)
# Returns (is_valid: bool, issues: list[str], details: ReviewQualityDetails)
is_valid, issues, details = validate_review_quality(
response=review_text,
review_type="executive_summary", # ReviewType
model_name="gemma3:4b",
)
```
`ReviewType` accepts: `"executive_summary"`, `"quality_review"`, `"methodology_review"`, `"improvement_suggestions"`, `"translation"`.
## Integration with the Review Pipeline
The full pipeline is orchestrated by `pipeline_runner.py`, invoked from `scripts/06_llm_review.py`. The canonical flow:
```python
from infrastructure.llm.review import select_and_start_ollama_model
from infrastructure.llm.review import create_review_client, warmup_model
from infrastructure.llm.review import extract_manuscript_text
from infrastructure.llm.review import generate_llm_executive_summary
from infrastructure.llm.review import generate_improvement_suggestions
from infrastructure.llm.review import generate_translation, save_review_outputs
from infrastructure.llm.review.generator import generate_quality_review
from infrastructure.llm.review.generator import generate_methodology_review
from infrastructure.llm.review.metrics import SessionMetrics, ManuscriptInputMetrics
from pathlib import Path
model_name = select_and_start_ollama_model()
client = create_review_client(model_name)
warmup_model(client)
text, input_metrics = extract_manuscript_text(
Path("output/project/pdf/combined.pdf")
)
reviews = {}
if text:
for fn, key in [
(generate_llm_executive_summary, "executive_summary"),
(generate_quality_review, "quality_review"),
(generate_methodology_review, "methodology_review"),
(generate_improvement_suggestions, "improvement_suggestions"),
]:
review_text, metrics = fn(client, text, model_name)
if review_text:
reviews[key] = review_text
save_review_outputs(
reviews=reviews,
output_dir=Path("output/project/llm"),
model_name=model_name,
pdf_path=Path("output/project/pdf/combined.pdf"),
session_metrics=SessionMetrics(),
)
```
## Configuration
Runtime behaviour is controlled by environment variables read by `OllamaClientConfig.from_env()`:
```bash
# Model selection
export OLLAMA_MODEL="gemma3:4b"
export OLLAMA_HOST="http://localhost:11434"
export OLLAMA_AUTO_START="true"
# Generation parameters
export LLM_LONG_MAX_TOKENS="16384"
export LLM_REVIEW_TIMEOUT="300" # seconds per review
export LLM_HEARTBEAT_INTERVAL="30"
```
## Testing
Follow the project no-mocks policy — use real function calls with `pytest-httpserver` or `ollama_test_server` for HTTP coverage:
```python
from infrastructure.llm.review.metrics import ReviewMetrics, estimate_tokens
from infrastructure.llm.review.quality import validate_review_quality
def test_estimate_tokens():
# estimate_tokens approximates at ~4 chars/token
assert estimate_tokens("hello world") == len("hello world") // 4
def test_validate_review_quality_short_response():
is_valid, issues, _ = validate_review_quality(
response="Too short.",
review_type="quality_review",
)
assert not is_valid
assert any("word" in i.lower() or "short" in i.lower() for i in issues)
```
## See Also
**Related Documentation:**
- [`../core/AGENTS.md`](../core/AGENTS.md) - LLM core client
- [`../templates/AGENTS.md`](../templates/AGENTS.md) - Template system
- [`../AGENTS.md`](../AGENTS.md) - LLM module overview
**System Documentation:**
- [`../../../AGENTS.md`](../../../AGENTS.md) - system overview
- [`../../../docs/operational/troubleshooting/llm-review.md`](../../../docs/operational/troubleshooting/llm-review.md) - LLM review troubleshooting