scrape-strategy ยท diff
v1.0.0 to git:20260327.66876fa
44 added, 94 removed. Audit A to C.
- ---
- name: scrape-strategy
- version: "1.0.0"
- description: "Web scraping strategy and troubleshooting. Use when scraping fails, site protection is encountered, or the user asks about scraping tier behavior."
-
- tools:
- - estimate_run
- - research_company
- - check_jobs
-
- resources:
- - file://references/tiers
-
- metadata:
- openclaw:
- requires:
- bins:
- - primr-mcp
- env:
- - GEMINI_API_KEY
- ---
-
- # Scrape Strategy
-
- Primr uses an 8-tier fallback system for web scraping. See `references/tiers.md` for the full tier table and selection heuristics.
-
- ## Key Features
-
- - **Sticky Tier**: Once a tier works for a host, it's tried first for subsequent pages
- - **Circuit Breaker**: After 3 consecutive failures of the same tier for a host, that tier is skipped
- - **Cookie Handoff**: Cookies obtained by browser tiers are reused by faster HTTP tiers
- - **Content Validation**: Checks actual content, not just HTTP status -- catches "200 OK" responses that are actually block pages
-
- ## Error Handling
-
- ### Content Validation Indicators
- - Content length < 1000 bytes
- - Contains "access denied", "blocked", "captcha"
- - Missing expected content markers
- - Redirect to login/challenge page
-
- ### Tier Escalation
- On failure: log reason, check circuit breaker, try next tier. Stops after 3 consecutive same-error failures.
-
- ### Recovery Strategies
-
- | Failure Type | Strategy |
- |--------------|----------|
- | Timeout | Increase timeout, try slower tier |
- | 403 Forbidden | Try stealth tier (4-5) |
- | 429 Rate Limit | Exponential backoff, reduce concurrency |
- | SSL Error | Try TLS compatibility tier (3) |
- | Empty Content | Try aggressive tier (2) |
- | CAPTCHA | Skip page, note in results |
-
- ## Interpreting Results
-
- ```
- + 34/46 pages scraped
- 34 = successfully scraped, 46 = total selected, 12 = failed
- ```
-
- - 70%+ success rate: Good coverage
- - 50-70%: Acceptable for protected sites
- - <50%: Consider deep mode instead
-
- ## Example Workflow
-
- ```
- User: "The site seems heavily protected"
-
- 1. Check prior scrape results:
- - Success rate: 35%
- - Most pages blocked at tier 4
-
- 2. Recommend strategy:
- "This site has strong protection. I recommend:
- - Use deep mode for external research
- - Or accept partial scrape results
-
- Deep mode gathers information from external sources
- without needing to access the protected site directly."
-
- 3. If user chooses deep mode:
- estimate_run(company, url, "deep")
- -> Cost: $0.80, Time: ~12 minutes
- ```
-
- ## Constraints
-
- - **Patient Timeout**: 90s max per page (allows multiple tier attempts)
- - **Concurrency**: 3 concurrent pages default
- - **Circuit Breaker**: 3 failures before tier skip
- - **Smart Escalation**: Stops after 3 consecutive same-error failures
+ ---
+ name: scrape-strategy
+ version: "1.1.0"
+ description: "Guide mode selection when site scraping is weak, blocked, or low quality. Use when the user asks why scraping failed or how Primr should proceed on a protected site."
+ mcp_server: "primr"
+ tools:
+ - estimate_run
+ - check_jobs
+ resources:
+ - primr://research/modes
+ - primr://research/status
+ - file://references/tiers
+ ---
+
+ # Scrape Strategy
+
+ ## Purpose
+
+ Use this skill when the bottleneck is site access rather than report writing. Focus on deciding whether to stay in scrape-capable modes or pivot to external research.
+
+ ## Workflow
+
+ 1. Read `primr://research/modes` to compare scrape, deep, full, and premium behavior.
+ 2. Review current run state through `primr://research/status` or `check_jobs`.
+ 3. Use `references/tiers.md` only when you need tier-level scraping context.
+ 4. Recommend the smallest viable change: retry, change mode, or accept partial coverage.
+
+ ## Decision Rules
+
+ - Prefer `deep` when the target site is heavily protected or first-party signal is sparse.
+ - Stay with `scrape` or `full` when first-party pages are the core evidence source.
+ - Do not promise exact success percentages unless the run data already shows them.
+ - Escalate from site troubleshooting to mode selection quickly; avoid over-explaining scrape internals unless the user asks.
+
+ ## Example
+
+ ```text
+ User: The site seems blocked
+
+ 1. Read primr://research/status
+ 2. Read primr://research/modes
+ 3. Explain whether deep mode is the better fit
+ 4. If needed, estimate_run(company_url="https://example.com", mode="deep")
+ ```