sciatlas-idea-evaluate ยท diff
git:20260616.7b5dc46 to git:20260713.1fd563d
93 added, 53 removed. Audit A to A.
---
name: sciatlas-idea-evaluate
- description: Use only SciAtlas search-papers to take a novice user from zero setup to an evidence-grounded evaluation of a research idea's novelty, feasibility, soundness, and differentiation. Trigger when the user asks whether an idea is worth pursuing or needs literature-backed critique.
+ description: Use only the current SciAtlas automated review workflow (`review_pipeline`) to take a novice user from zero setup to a final automated review of a research idea or paper, including setup, registration guidance, workflow configuration, retrieval, artifact reading, and synthesis. Trigger when the user asks whether an idea is worth pursuing, wants automatic review, meta-review, rubric-based critique, novelty/feasibility/soundness assessment, or literature-backed evaluation.
---
# SciAtlas Idea Evaluate
- Use this skill to evaluate a research idea with evidence from `search-papers` only.
+ Use this skill to run the repository automated review workflow. The workflow builds idea context, searches KG/S2 evidence, creates a manifest, grounds claims to paper paragraphs, builds rubric evidence, samples reviewer backgrounds, generates reviewer reports, and synthesizes a final report.
## Operating Contract
- - Do not call any SciAtlas downstream command. The only SciAtlas retrieval command allowed is `search-papers`.
- - Ask the user only for email, verification code, token, or one necessary clarification.
- - Run setup and retrieval yourself where possible.
- - Do not expose the full token.
+ - Run only `sciatlas idea-evaluate` or `python run_sciatlas.py idea-evaluate` for this skill.
+ - Own the end-to-end novice flow: install or locate the CLI, guide registration, configure `.env` or shell variables, run the workflow, inspect artifacts, and synthesize the final review result.
+ - Ask the user only for human-only values: missing idea/PDF path, email, verification code, SciAtlas token, LLM/S2/KG credentials that are not already configured, or one necessary scope clarification.
+ - Do not ask the user to run shell commands when tool access is available.
+ - Use `--workflow flash` by default.
+ - Use `--workflow full` for a broader reviewer/rubric/evidence pass or when the user wants a more comprehensive review.
+ - Never disclose full API keys or tokens.
+ - Read saved artifacts before answering.
## Zero-Start Bootstrap
- 1. Check whether the `sciatlas` executable exists without invoking a SciAtlas command: use `Get-Command sciatlas -ErrorAction SilentlyContinue` on Windows PowerShell or `command -v sciatlas` on macOS/Linux. Install if missing with `python -m pip install -e ./sciatlas` or `python -m pip install "git+https://github.com/zjunlp/SciAtlas.git#subdirectory=sciatlas"`.
- 2. If no `SCIATLAS_API_KEY` is available, open or provide `http://sciatlas.openkg.cn/register`.
- 3. Walk the novice user through registration. Ask for the email verification code when required and the returned `sciatlas_xxx` token when registration finishes.
- 4. Configure the shell:
+ 1. Check whether the repository command works:
- ```powershell
- $env:SCIATLAS_API_BASE_URL = "http://sciatlas.openkg.cn"
- $env:SCIATLAS_API_KEY = "<token>"
- setx SCIATLAS_API_BASE_URL "http://sciatlas.openkg.cn"
- setx SCIATLAS_API_KEY "<token>"
+ ```bash
+ python run_sciatlas.py idea-evaluate -h
```
+ If needed, fall back to `sciatlas idea-evaluate -h` after installing the full checkout.
+
+ 2. This dedicated workflow requires a full SciAtlas checkout. If it is missing, clone the repository, change into it, then run `python -m pip install -e ./sciatlas` and `python -m pip install -r requirements-workflows.txt`. Do not use the GitHub `#subdirectory=sciatlas` package-only installation for this workflow.
+ 3. Check current environment and `.env` for `SCIATLAS_API_KEY`, LLM settings, S2 settings, and KG settings before asking the user.
+ 4. If no SciAtlas token is configured, guide the user to `http://sciatlas.openkg.cn/register`; ask for email, verification code, and returned `sciatlas_xxx` token only when needed.
+ 5. If LLM/S2/KG credentials are required and missing, ask only for the missing values. Use the user's provider values without printing them back.
+ 6. Configure the current shell or `.env` yourself, then run the workflow.
+
+ Configure workflow credentials in `.env` or the shell:
+
```bash
- export SCIATLAS_API_BASE_URL="http://sciatlas.openkg.cn"
- export SCIATLAS_API_KEY="<token>"
+ SCIATLAS_API_BASE_URL=http://sciatlas.openkg.cn
+ SCIATLAS_API_KEY=<sciatlas-token>
+ NEO4J_URI=<local-or-hosted-neo4j-uri>
+ NEO4J_USER=<neo4j-user>
+ NEO4J_PASSWORD=<neo4j-password>
+ S2_API_KEY=<semantic-scholar-key>
+ OPENAI_API_KEY=<llm-key>
+ OPENAI_BASE_URL=https://api.deepseek.com
+ LLM_MODEL=deepseek-v4-flash
```
- 5. Use only `search-papers` for any token smoke test.
-
- ## Evaluation Frame
+ The workflow also accepts `DMX-API-KEY`, `DMX_API_KEY`, `LLM_API_KEY`, `LLM_BASE_URL`, `LLM_API_URL`, `SEARCH_LLM_API_KEY`, `SEARCH_LLM_API_URL`, and `SEARCH_LLM_MODEL`.
- Convert the idea into four claims:
+ ## Run Plan
- - Novelty claim: what is supposed to be new.
- - Feasibility claim: why it can be built with available methods/data.
- - Soundness claim: why the mechanism should work.
- - Differentiation claim: why it is not just a minor variant.
+ Flash path for an idea:
- ## Search Plan
+ ```bash
+ python run_sciatlas.py idea-evaluate \
+ --idea "<research idea>" \
+ --workflow flash
+ ```
- Run only `search-papers`. Use one direct search and, if needed, one stress-test search:
+ Full path:
```bash
- sciatlas search-papers --retrieval-mode hybrid --query "<research idea>" --keyword "high:<core mechanism>" --keyword "middle:<application domain>" --top-k 8 --top-keywords 0 --max-titles 0 --max-refs 0 --bias-keyword high --bias-related high --bias-citation low --bias-exploration low --ranking-profile precision --report-max-items 8
+ python run_sciatlas.py idea-evaluate \
+ --idea "<research idea>" \
+ --workflow full
```
- Stress-test novelty by searching for the closest alternative wording:
+ For a paper PDF:
```bash
- sciatlas search-papers --retrieval-mode hybrid --query "<closest competing formulation>" --keyword "high:<problem>" --keyword "middle:<neighbor method>" --top-k 8 --top-keywords 0 --max-titles 0 --max-refs 0 --bias-related high --ranking-profile precision --report-max-items 8
+ python run_sciatlas.py idea-evaluate \
+ --pdf path/to/paper.pdf \
+ --workflow full
```
- ## Reading Artifacts
+ Useful overrides:
- Read `report.md` and `response.json`. Build an evaluation ledger:
+ - `--top-k N`, `--kg-top-k N`, `--s2-top-k N`, `--manifest-top-k N` tune evidence breadth.
+ - `--max-reviewers N` controls reviewer fan-out.
+ - `--grounding-final-top-k N` controls paragraph evidence depth.
+ - `--short-report` requests compact meta-review output in full mode.
+ - `--disable-review-llm` uses deterministic fallback when LLM review is unavailable.
+ - `--smoke` runs a stubbed pipeline for structural validation.
- - Evidence for novelty.
- - Evidence against novelty.
- - Evidence for feasibility: data, implementation pattern, benchmark, or system precedent.
- - Evidence for soundness: theory, empirical result, mechanism, or ablation.
- - Missing evidence.
+ ## Workflow Modes
- ## Evaluation Method
+ `flash` compresses nonessential stages:
- Score each dimension as High, Medium, Low, or Unknown:
+ - smaller KG/S2/manifest budgets;
+ - one reviewer by default;
+ - compact evidence-card budgets;
+ - grounding refinement disabled;
+ - short meta-review enabled.
- 1. Novelty: High only if close papers solve a different problem or use a meaningfully different mechanism.
- 2. Feasibility: High only if retrieved work shows required components, data, or evaluation are realistic.
- 3. Soundness: High only if evidence supports the core causal mechanism, not merely the topic.
- 4. Differentiation: High only if the idea can be expressed as a clear contrast with closest prior work.
+ `full` keeps the comprehensive path:
- For each score:
+ - broader KG/S2/manifest budgets;
+ - reviewer fan-out and reviewer-background branch;
+ - rubric-source and rubric-LLM branches;
+ - paragraph grounding, reviewer reports, and full report synthesis.
- - cite the supporting papers from the search result;
- - explain the reasoning in plain language;
- - state the biggest uncertainty;
- - suggest one concrete improvement.
+ ## Artifacts To Read
+ Read the run directory:
+
+ - `summary.json`: wrapper status, workflow mode, subprocess command, and artifact pointers.
+ - `result.json`: review pipeline final payload and stage statuses.
+ - `report.md`: user-facing final report or compact meta-review.
+ - `idea_context/idea_context.json`: normalized idea/PDF context and structured idea extraction.
+ - `search/result.json`: KG/S2 search evidence.
+ - `manifest/manifest.json`: papers selected for paragraph extraction.
+ - `grounding/result.json`: paragraph grounding and experiment grounding.
+ - `rubric_sources/result.json`, `rubric_llm/result.json`: rubric evidence when available.
+ - `review/reviewer_reviews.index.json`: reviewer-level evaluations.
+ - `report/final_report.md`, `report/final_report.json`: native final outputs.
+ - `logs/*.log` and `logs/review_pipeline.*.txt`: stage logs when anything is partial or failed.
+
+ Treat `partial_error` as usable when the final report exists, but disclose which branch failed or was skipped.
+
## Deliverable
Return:
- - Search command(s), with token omitted.
- - Evaluation table: novelty, feasibility, soundness, differentiation.
- - Closest-prior-art risk.
- - Go / revise / no-go recommendation.
- - Revised idea statement and next `search-papers` query.
+ - exact command used, with credentials omitted;
+ - workflow mode and artifact paths;
+ - overall go/revise/no-go recommendation;
+ - novelty, feasibility, soundness, and differentiation assessment;
+ - closest-prior-art risk;
+ - strongest supporting evidence and biggest missing evidence;
+ - concrete revision advice and next SciAtlas query.
+
+ Keep judgments tied to retrieved papers, grounding snippets, reviewer reports, or rubric artifacts.