predictor-hand-skill · v1.0.0 · 2026-07-10 · sha256 7799ce7930d197df
predictor-hand-skill v1.0.0A
Immutable. This exact content is served forever at /api/v1/blob/7799ce7930d197df.
---
name: predictor-hand-skill
version: "1.0.0"
description: "Expert knowledge for AI forecasting — superforecasting principles, signal taxonomy, confidence calibration, reasoning chains, and accuracy tracking"
runtime: prompt_only
---
# Forecasting Expert Knowledge
## Superforecasting Principles
Based on research by Philip Tetlock and the Good Judgment Project:
1. **Triage**: Focus on questions that are hard enough to be interesting but not so hard they're unknowable
2. **Break problems apart**: Decompose big questions into smaller, researchable sub-questions (Fermi estimation)
3. **Balance inside and outside views**: Use both specific evidence AND base rates from reference classes
4. **Update incrementally**: Adjust predictions in small steps as new evidence arrives (Bayesian updating)
5. **Look for clashing forces**: Identify factors pulling in opposite directions
6. **Distinguish signal from noise**: Weight signals by their reliability and relevance
7. **Calibrate**: Your 70% predictions should come true ~70% of the time
8. **Post-mortem**: Analyze why predictions went wrong, not just celebrate the right ones
9. **Avoid the narrative trap**: A compelling story is not the same as a likely outcome
10. **Collaborate**: Aggregate views from diverse perspectives
---
## Signal Taxonomy
### Signal Types
| Type | Description | Weight | Example |
|------|-----------|--------|---------|
| Leading indicator | Predicts future movement | High | Job postings surge → company expanding |
| Lagging indicator | Confirms past movement | Medium | Quarterly earnings → business health |
| Base rate | Historical frequency | High | "80% of startups fail within 5 years" |
| Expert opinion | Informed prediction | Medium | Analyst forecast, CEO statement |
| Data point | Factual measurement | High | Revenue figure, user count, benchmark |
| Anomaly | Deviation from pattern | High | Unusual trading volume, sudden hiring freeze |
| Structural change | Systemic shift | Very High | New regulation, technology breakthrough |
| Sentiment shift | Collective mood change | Medium | Media tone change, social media trend |
### Signal Strength Assessment
```
STRONG signal (high predictive value):
- Multiple independent sources confirm
- Quantitative data (not just opinions)
- Leading indicator with historical track record
- Structural change with clear causal mechanism
MODERATE signal (some predictive value):
- Single authoritative source
- Expert opinion from domain specialist
- Historical pattern that may or may not repeat
- Lagging indicator (confirms direction)
WEAK signal (limited predictive value):
- Social media buzz without substance
- Single anecdote or case study
- Rumor or unconfirmed report
- Opinion from non-specialist
```
---
## Confidence Calibration
### Probability Scale
```
95% — Almost certain (would bet 19:1)
90% — Very likely (would bet 9:1)
80% — Likely (would bet 4:1)
70% — Probable (would bet 7:3)
60% — Slightly more likely than not
50% — Toss-up (genuine uncertainty)
40% — Slightly less likely than not
30% — Unlikely (but plausible)
20% — Very unlikely (but possible)
10% — Extremely unlikely
5% — Almost impossible (but not zero)
```
### Calibration Rules
1. NEVER use 0% or 100% — nothing is absolutely certain
2. If you haven't done research, default to the base rate (outside view)
3. Your first estimate should be the reference class base rate
4. Adjust from the base rate using specific evidence (inside view)
5. Typical adjustment: ±5-15% per strong signal, ±2-5% per moderate signal
6. If your gut says 80% but your analysis says 55%, trust the analysis
### Brier Score
The gold standard for measuring prediction accuracy:
```
Brier Score = (predicted_probability - actual_outcome)^2
actual_outcome = 1 if prediction came true, 0 if not
Perfect score: 0.0 (you're always right with perfect confidence)
Coin flip: 0.25 (saying 50% on everything)
Terrible: 1.0 (100% confident, always wrong)
Good forecaster: < 0.15
Average forecaster: 0.20-0.30
Bad forecaster: > 0.35
```
---
## Domain-Specific Source Guide
### Technology Predictions
| Source Type | Examples | Use For |
|-------------|---------|---------|
| Product roadmaps | GitHub issues, release notes, blog posts | Feature predictions |
| Adoption data | Stack Overflow surveys, NPM downloads, DB-Engines | Technology trends |
| Funding data | Crunchbase, PitchBook, TechCrunch | Startup success/failure |
| Patent filings | Google Patents, USPTO | Innovation direction |
| Job postings | LinkedIn, Indeed, Levels.fyi | Technology demand |
| Benchmark data | TechEmpower, MLPerf, Geekbench | Performance trends |
### Finance Predictions
| Source Type | Examples | Use For |
|-------------|---------|---------|
| Economic data | FRED, BLS, Census | Macro trends |
| Earnings | SEC filings, earnings calls | Company performance |
| Analyst reports | Bloomberg, Reuters, S&P | Market consensus |
| Central bank | Fed minutes, ECB statements | Interest rates, policy |
| Commodity data | EIA, OPEC reports | Energy/commodity prices |
| Sentiment | VIX, put/call ratio, AAII survey | Market mood |
### Geopolitics Predictions
| Source Type | Examples | Use For |
|-------------|---------|---------|
| Official sources | Government statements, UN reports | Policy direction |
| Think tanks | RAND, Brookings, Chatham House | Analysis |
| Election data | Polls, voter registration, 538 | Election outcomes |
| Trade data | WTO, customs data, trade balances | Trade policy |
| Military data | SIPRI, defense budgets, deployments | Conflict risk |
| Diplomatic signals | Ambassador recalls, sanctions, treaties | Relations |
### Climate Predictions
| Source Type | Examples | Use For |
|-------------|---------|---------|
| Scientific data | IPCC, NASA, NOAA | Climate trends |
| Energy data | IEA, EIA, IRENA | Energy transition |
| Policy data | COP agreements, national plans | Regulation |
| Corporate data | CDP disclosures, sustainability reports | Corporate action |
| Technology data | BloombergNEF, patent filings | Clean tech trends |
| Investment data | Green bond issuance, ESG flows | Capital allocation |
---
## Reasoning Chain Construction
### Template
```
PREDICTION: [Specific, falsifiable claim]
1. REFERENCE CLASS (Outside View)
Base rate: [What % of similar events occur?]
Reference examples: [3-5 historical analogues]
2. SPECIFIC EVIDENCE (Inside View)
Signals FOR (+):
a. [Signal] — strength: [strong/moderate/weak] — adjustment: +X%
b. [Signal] — strength: [strong/moderate/weak] — adjustment: +X%
Signals AGAINST (-):
a. [Signal] — strength: [strong/moderate/weak] — adjustment: -X%
b. [Signal] — strength: [strong/moderate/weak] — adjustment: -X%
3. SYNTHESIS
Starting probability (base rate): X%
Net adjustment: +/-Y%
Final probability: Z%
4. KEY ASSUMPTIONS
- [Assumption 1]: If wrong, probability shifts to [W%]
- [Assumption 2]: If wrong, probability shifts to [V%]
5. RESOLUTION
Date: [When can this be resolved?]
Criteria: [Exactly how to determine if correct]
Data source: [Where to check the outcome]
```
---
## Worked Examples
### Example 1: Corporate Acquisition
**Question**: "Will Acme Corp be acquired within 12 months?" (asked January 2025)
```
PREDICTION: Acme Corp (mid-cap SaaS, $2B market cap) will be acquired by January 2026
1. REFERENCE CLASS (Outside View)
Base rate: ~5-7% of publicly traded mid-cap SaaS companies receive
acquisition offers in any given 12-month period.
Reference examples:
- Splunk acquired by Cisco (2023) — similar scale, strategic buyer
- Figma attempted acquisition by Adobe (2022) — regulatory block
- Nuance acquired by Microsoft (2021) — vertical SaaS, strategic fit
- Mandiant acquired by Google (2022) — security vertical
- Cvent acquired by Blackstone (2021) — PE buyout at depressed valuation
Starting probability: 6%
2. SPECIFIC EVIDENCE (Inside View)
Signals FOR (+):
a. Board hired Goldman Sachs as advisor (leaked filing)
— strength: STRONG — adjustment: +20%
(Companies that retain M&A advisors complete a transaction ~40% of the time)
b. CEO sold 30% of personal holdings in Q4 (SEC filing)
— strength: MODERATE — adjustment: +5%
c. Two major competitors acquired in past 18 months (market consolidation)
— strength: MODERATE — adjustment: +8%
d. Revenue growth decelerated from 35% to 18% YoY (earnings report)
— strength: MODERATE — adjustment: +5%
(Slower-growth companies more likely to accept acquisition offers)
Signals AGAINST (-):
a. Founder still holds 25% voting control and has said "we're building for
the long term" (recent interview)
— strength: STRONG — adjustment: -12%
b. Stock price at all-time high — acquirer must pay steep premium
— strength: MODERATE — adjustment: -5%
c. Current antitrust environment — FTC blocking more deals
— strength: WEAK — adjustment: -3%
3. SYNTHESIS
Starting probability (base rate): 6%
Signals for: +20% +5% +8% +5% = +38%
Signals against: -12% -5% -3% = -20%
Net adjustment: +18%
Raw probability: 24%
Sanity check: ~1 in 4 feels right given the strong M&A advisor signal
balanced against founder control.
Final probability: 25%
4. KEY ASSUMPTIONS
- Goldman engagement is for M&A (not debt restructuring):
If wrong, probability drops to 8%
- Founder is willing to sell at the right price:
If wrong (founder vetoes any deal), probability drops to 3%
- Regulatory environment doesn't tighten further:
If wrong, probability drops to 18%
5. RESOLUTION
Date: January 31, 2026
Criteria: Definitive merger agreement announced (not just rumors)
Data source: SEC EDGAR (8-K filing), Bloomberg terminal
```
### Example 2: Technology Adoption
**Question**: "Will WebAssembly (Wasm) reach mainstream server-side adoption by 2027?"
```
PREDICTION: >20% of new cloud-deployed services will use Wasm runtimes by
end of 2027
1. REFERENCE CLASS (Outside View)
Technology adoption lifecycle (Rogers curve):
- Innovators (2.5%) → Early Adopters (13.5%) → Early Majority (34%)
- Crossing from Early Adopters to Early Majority typically takes 3-5 years
after first production deployments
- First serious server-side Wasm deployments: ~2022 (Fermyon, Cosmonic)
- Current status (2025): Late Early Adopter stage
Historical analogues for infrastructure tech adoption:
- Containers (Docker 2013 → mainstream 2017-2018): ~4-5 years
- Kubernetes (2014 → mainstream 2018-2019): ~4-5 years
- Serverless (Lambda 2014 → mainstream 2018-2020): ~4-6 years
Base rate for "infrastructure tech reaching 20% adoption within 5 years
of first production use": ~30%
Starting probability: 30%
2. SPECIFIC EVIDENCE (Inside View)
Signals FOR (+):
a. WASI standard maturing — WASI Preview 2 shipped, component model
stabilizing (W3C working group)
— strength: STRONG — adjustment: +8%
b. Major cloud providers offering Wasm runtimes (Fastly, Cloudflare Workers,
Azure, AWS exploring)
— strength: STRONG — adjustment: +10%
c. Docker adding Wasm support natively (announced 2022, shipping)
— strength: MODERATE — adjustment: +5%
Signals AGAINST (-):
a. Ecosystem still fragmented — multiple competing runtimes, toolchain gaps
— strength: STRONG — adjustment: -10%
b. Containers already "good enough" for most workloads — weak forcing
function to switch
— strength: STRONG — adjustment: -8%
c. Wasm language support uneven — great for Rust/C++, mediocre for Python/JS
— strength: MODERATE — adjustment: -5%
Leading indicators to track:
- CNCF survey: % of respondents evaluating/using Wasm
- Job postings mentioning Wasm (Indeed/LinkedIn trend)
- GitHub stars and contributors for top Wasm runtimes (wasmtime, wasmer)
- WASI spec milestone dates vs planned dates
3. SYNTHESIS
Starting probability (base rate): 30%
Signals for: +8% +10% +5% = +23%
Signals against: -10% -8% -5% = -23%
Net adjustment: 0%
Final probability: 30%
Interpretation: The positive and negative signals roughly cancel out.
The base rate from analogous infrastructure technologies holds.
This is genuinely uncertain — the "chasm" crossing is the key risk.
4. KEY ASSUMPTIONS
- "Mainstream" defined as >20% of NEW deployments (not total installed base)
- WASI component model reaches 1.0 stable by mid-2026
If delayed beyond 2026: probability drops to 15%
- No competing paradigm emerges (e.g., eBPF expanding scope):
If strong competitor: probability drops to 20%
5. RESOLUTION
Date: December 31, 2027
Criteria: CNCF annual survey shows >20% respondents using Wasm in production
Data source: CNCF Annual Survey, Datadog Container Report
```
### Example 3: Geopolitical Forecast
**Question**: "Will US-China trade tensions escalate significantly in 2025?"
(Defined as: new tariffs >25% on >$100B of goods, or export controls expanded
to 3+ new technology categories)
```
PREDICTION: Significant escalation of US-China trade tensions in 2025
1. REFERENCE CLASS (Outside View)
Historical trade conflict escalation pattern:
- US-China trade relations since 2018: escalation occurred in 4 of 7 years
- In election year +1 (new/returning administration): escalation rate ~60%
- Trade wars historically escalate in steps, with retaliation cycles
Starting probability: 55%
2. SPECIFIC EVIDENCE (Inside View)
Signals FOR (+):
a. Administration rhetoric on China hawkish across both parties
— strength: STRONG — adjustment: +10%
b. Semiconductor export controls already expanding (ASML, Tokyo Electron)
— strength: STRONG — adjustment: +8%
c. China retaliating with rare earth export restrictions
— strength: MODERATE — adjustment: +5%
Signals AGAINST (-):
a. Business lobbying against further tariffs (Chamber of Commerce, farm lobby)
— strength: MODERATE — adjustment: -5%
b. Inflation concerns create political cost for tariffs
— strength: MODERATE — adjustment: -5%
c. Diplomatic channels active (recent bilateral meetings)
— strength: WEAK — adjustment: -3%
Scenario mapping:
┌─────────────────────────┬─────────────┬────────────────────┐
│ Scenario │ Probability │ Key trigger │
├─────────────────────────┼─────────────┼────────────────────┤
│ Major escalation │ 25% │ Taiwan crisis or │
│ (new tariffs + controls │ │ tech IP theft case │
│ + retaliatory cycle) │ │ │
├─────────────────────────┼─────────────┼────────────────────┤
│ Moderate escalation │ 40% │ Incremental tariff │
│ (meets our threshold) │ │ increases + 1-2 │
│ │ │ new export controls│
├─────────────────────────┼─────────────┼────────────────────┤
│ Status quo / minor │ 30% │ Diplomatic deals, │
│ changes │ │ election distraction│
├─────────────────────────┼─────────────┼────────────────────┤
│ De-escalation │ 5% │ Grand bargain │
│ (reduced tariffs) │ │ (historically rare)│
└─────────────────────────┴─────────────┴────────────────────┘
P(meets our escalation threshold) = 25% + 40% = 65%
3. SYNTHESIS
Starting probability (base rate): 55%
Signals for: +10% +8% +5% = +23%
Signals against: -5% -5% -3% = -13%
Net adjustment: +10%
Raw probability: 65%
Cross-check with scenario mapping: 65% — consistent.
Final probability: 65%
4. KEY ASSUMPTIONS
- No major geopolitical crisis (Taiwan strait) that causes extreme
escalation or extreme restraint: If crisis occurs, split to
80% (escalation) or 20% (restraint/avoidance)
- US economy remains stable: If recession hits, probability drops
to 45% (political cost of tariffs rises)
- China does not make major trade concessions preemptively:
If it does, probability drops to 30%
5. RESOLUTION
Date: December 31, 2025
Criteria: Cumulative new tariffs >25% on >$100B goods OR export controls
expanded to 3+ new technology categories (per USTR/BIS announcements)
Data source: USTR tariff schedule, BIS Entity List updates, Congressional
Research Service reports
```
---
## Fermi Estimation Techniques
Fermi estimation is the art of making reasonable order-of-magnitude guesses
by breaking unknowable questions into smaller, estimable pieces.
### Step-by-Step Process
```
1. DEFINE the quantity you want to estimate
→ Be specific about units, scope, and timeframe
2. DECOMPOSE into factors you can estimate independently
→ Prefer multiplication chains: A × B × C
→ Each factor should be something you can reason about
3. ESTIMATE each factor
→ Use round numbers (powers of 10 when possible)
→ State your confidence range for each factor
4. MULTIPLY and sanity-check
→ Does the result pass the "smell test"?
→ Cross-check with any known anchors
5. STATE your uncertainty
→ Fermi estimates are typically accurate within 1 order of magnitude
→ Give a range: [estimate / 3, estimate × 3] is a reasonable default
```
### Common Reference Anchors
Keep these memorized for quick estimation:
```
POPULATION
World: ~8 billion
US: ~340 million
EU: ~450 million
China: ~1.4 billion
India: ~1.4 billion
ECONOMICS
World GDP: ~$100 trillion
US GDP: ~$28 trillion
US median household: ~$75,000/year
US federal budget: ~$6.5 trillion
S&P 500 total cap: ~$45 trillion
TIME
Seconds in a day: ~86,400 (~10^5)
Seconds in a year: ~31.5 million (~3 × 10^7)
Working hours/year: ~2,000
TECHNOLOGY
Global internet users: ~5.5 billion
Global smartphone users: ~4.5 billion
AWS annual revenue: ~$90 billion
Global IT spending: ~$5 trillion
GitHub developers: ~100 million
INDUSTRY SIZES (annual, global)
Cloud computing: ~$600 billion
Semiconductor: ~$600 billion
Pharmaceutical: ~$1.5 trillion
Automotive: ~$3 trillion
Agriculture: ~$3 trillion
E-commerce: ~$6 trillion
```
### Worked Fermi Examples
**Example A: Estimating the TAM for an AI code review tool**
```
Question: What is the annual TAM for an AI-powered code review SaaS?
Decomposition:
TAM = (Number of professional developers)
× (% who do code reviews regularly)
× (willingness to pay for tooling)
× (average annual price)
Estimates:
Professional developers worldwide: ~30 million
(GitHub has 100M accounts, but ~30% are professional, and
not all professionals use GitHub)
% who do code reviews: ~60%
(Standard in companies > 50 engineers, less common in small shops)
Target market (teams that would buy SaaS): ~40%
(Enterprise and mid-market; small teams use free tools)
Annual price per seat: ~$300/year
(Comparable: GitHub Copilot ~$200, Snyk ~$400, middle ground)
Calculation:
30M × 0.60 × 0.40 × $300 = $2.16 billion
Sanity check:
- GitHub revenue ~$2B (broader product, ~4M paid users)
- Snyk valued at $7B (code security, related space)
- $2B TAM is plausible for a focused code review tool
Result: ~$2 billion TAM (range: $700M to $6B)
```
**Example B: Estimating daily active queries to a search engine**
```
Question: How many search queries does Google process per day?
Decomposition:
Queries/day = (Internet users who use Google)
× (searches per user per day)
Estimates:
Global internet users: ~5.5 billion
Google market share: ~90%
Google users: 5.5B × 0.90 = ~5 billion
But not all use it daily: ~50% daily active rate
Daily active Google searchers: ~2.5 billion
Searches per active user per day: ~3-4
(Some people search 10+ times, many search once or not at all)
Calculation:
2.5 billion × 3.5 = ~8.5 billion queries/day
Sanity check:
Published figure (Google): ~8.5 billion searches/day (2024)
Our estimate nailed it — sometimes Fermi estimation gets lucky.
Result: ~8.5 billion/day (range: 3B to 25B)
```
### Order of Magnitude Sanity Checks
After any estimate, verify it makes sense:
```
CHECK 1: Per-person reasonableness
Divide by relevant population. Is the per-person number realistic?
"$50B market ÷ 340M Americans = $147/person" — plausible?
CHECK 2: Comparison to known quantities
Is your estimate bigger or smaller than things you know?
"Our estimate of X is $3B — that's 5% of AWS revenue. Reasonable?"
CHECK 3: Growth rate implied
If you're estimating a future state, what annual growth rate is implied?
>50% sustained growth for >3 years is extremely rare.
CHECK 4: Upper bound test
What is the theoretical maximum? Is your estimate within it?
"Total possible customers × maximum price = ceiling"
```
---
## Prediction Market Patterns
### Interpreting Market Prices as Probabilities
Prediction market prices map to probabilities, but with important caveats:
```
Market price $0.65 for "Event X occurs"
→ Naive interpretation: 65% probability
→ Adjusted interpretation: depends on market quality
Adjustment factors:
Liquid market (Polymarket, Metaculus with many forecasters):
Price ≈ true probability (±3-5%)
Thin market (<50 traders, <$10K volume):
Price is noisy — treat as ±15% uncertainty
A $0.65 price could represent 50-80% true probability
Binary vs. multi-outcome:
Binary markets are more reliable
Multi-outcome markets often have probabilities summing to >100%
(overround) — normalize before interpreting
```
### Common Prediction Market Biases
| Bias | Description | Impact | Correction |
|------|-------------|--------|------------|
| Favorite-longshot | Favorites underpriced, longshots overpriced | Longshot events appear ~2-3x more likely than they are | If market says 5%, true probability may be 2-3% |
| Recency | Recent events dominate pricing | Probability spikes after news, then slowly reverts | Wait 24-48h after major news before trusting market prices |
| Liquidity premium | Illiquid contracts trade at a discount | Prices biased toward 50% in thin markets | Weight liquid markets more heavily |
| Expiration clustering | Prices converge to 0 or 1 near expiration | Mid-probability contracts vanish near deadline | Most useful signal is months before resolution |
| Hedging distortion | Traders hedging other positions, not expressing beliefs | Prices reflect risk management, not pure probability | Cross-reference with non-market forecasts |
### Aggregation Methods
When combining multiple probability estimates (markets, experts, models):
```
SIMPLE AVERAGE
P = (P1 + P2 + P3) / 3
Use when: Sources are roughly equally credible
Weakness: Susceptible to outliers
MEDIAN
P = middle value of sorted estimates
Use when: One source might be badly miscalibrated
Weakness: Ignores magnitude of disagreement
TRIMMED MEAN
Drop highest and lowest, average the rest
P = average(P2 ... Pn-1) after sorting
Use when: 5+ sources, want outlier robustness
EXTREMIZED AVERAGE
P_avg = simple average
P_extremized = P_avg^a / (P_avg^a + (1-P_avg)^a), where a > 1
Typical a = 1.5 to 2.5 (more extremizing with more independent sources)
Use when: Sources are genuinely independent (not reading each other)
Rationale: If 5 independent sources all say 70%, the true probability
is likely higher than 70% — shared info should push further from 50%
CONFIDENCE-WEIGHTED AVERAGE
P = Σ(wi × Pi) / Σ(wi)
where wi = track record score or source reliability
Use when: Sources have known, differing track records
```
### When Markets Beat Experts (and Vice Versa)
```
MARKETS TEND TO WIN when:
✓ Large, liquid, diverse participant pool
✓ Question is well-defined with clear resolution criteria
✓ Information is widely distributed (no single expert has edge)
✓ Time horizon is 1 month to 2 years
Examples: Election outcomes, product launch dates, economic indicators
EXPERTS TEND TO WIN when:
✓ Question requires deep domain-specific knowledge
✓ Market is thin or participants lack domain context
✓ Very long time horizons (>5 years) — markets discount distant futures
✓ Novel situations with no historical market precedent
Examples: Technical feasibility, scientific breakthroughs, niche regulation
BEST PRACTICE: Use both
Start with the market price, then adjust using expert insight.
Treat the market as the prior and expert analysis as an update.
```
---
## Update Protocol
### Bayesian Updating Worked Example
```
SCENARIO: You predicted 30% chance that Company Z launches Product A in Q1.
New evidence: A leaked internal slide shows a Q1 launch timeline.
STEP 1: State the prior
P(launch in Q1) = 0.30
STEP 2: Assess the evidence
E = leaked slide showing Q1 timeline
How likely is this evidence if the launch IS happening in Q1?
P(E | launch) = 0.85
(Internal slides usually reflect real plans, but plans change)
How likely is this evidence if the launch is NOT in Q1?
P(E | no launch) = 0.15
(Could be outdated slide, aspirational, or decoy)
STEP 3: Calculate the likelihood ratio
LR = P(E | launch) / P(E | no launch) = 0.85 / 0.15 = 5.67
STEP 4: Convert prior to odds, multiply, convert back
Prior odds = 0.30 / 0.70 = 0.429
Posterior odds = 0.429 × 5.67 = 2.43
Posterior probability = 2.43 / (1 + 2.43) = 0.71
STEP 5: State the update
Prior: 30% → Posterior: 71%
Update magnitude: +41 percentage points
This is a LARGE update, appropriate because the evidence (internal
planning document) is strong and directly relevant.
```
### Evidence Strength Classification
How much to update based on different types of evidence:
```
EVIDENCE TIER 1 — Large update (likelihood ratio 5-20x)
→ Official announcement or regulatory filing
→ Confirmed internal document (not rumor)
→ Directly observed outcome of prerequisite event
→ Multiple independent strong sources confirming same fact
Typical update: ±15-30 percentage points
EVIDENCE TIER 2 — Moderate update (likelihood ratio 2-5x)
→ Credible journalist report with named sources
→ Statistical data that changes the base rate
→ Expert with strong track record changing their view
→ Structural/policy change that alters incentives
Typical update: ±5-15 percentage points
EVIDENCE TIER 3 — Small update (likelihood ratio 1.2-2x)
→ Rumor from semi-credible source
→ Anecdotal evidence (single data point)
→ Social media sentiment shift
→ Expert opinion without new information
Typical update: ±2-5 percentage points
EVIDENCE TIER 4 — Negligible update (likelihood ratio ~1x)
→ Repetition of previously known information
→ Pundit opinion with no domain expertise
→ Vague statement open to multiple interpretations
→ Evidence equally consistent with both outcomes
Typical update: ±0-2 percentage points (or skip entirely)
```
### When to Make Large vs. Small Updates
```
MAKE A LARGE UPDATE when:
• Evidence directly addresses your key uncertainty
• The source has a strong track record on this topic
• The evidence would be very surprising if your prediction were correct
(or very unsurprising if it were wrong)
• Multiple independent signals shift in the same direction simultaneously
MAKE A SMALL UPDATE when:
• Evidence is tangentially related to your prediction
• The source's reliability is uncertain
• The evidence is consistent with multiple interpretations
• You've already incorporated similar evidence
RESIST UPDATING when:
• The "evidence" is just someone restating the consensus
• A vivid anecdote feels compelling but carries no statistical weight
• You're reacting emotionally (fear, excitement) rather than analytically
• The evidence source has an obvious incentive to mislead
```
### Common Updating Mistakes
| Mistake | Description | Fix |
|---------|-------------|-----|
| Over-updating on vivid events | A dramatic single event shifts your view by 20+ points when the base rate barely moved | Ask: "Does this event actually change the base rate, or just my emotional state?" |
| Under-updating on base rate changes | New data shows the reference class frequency shifted, but you keep your old anchor | Periodically re-derive the base rate from scratch instead of only adjusting incrementally |
| Asymmetric updating | Updating strongly on confirming evidence, weakly on disconfirming evidence | Force yourself to calculate the likelihood ratio for disconfirming evidence explicitly |
| Double-counting | Updating on a news article, then updating again on a tweet quoting the same article | Track the original source — if two signals share the same root cause, count once |
| Failure to update | Knowing the evidence should change your view but not bothering because your current number "feels right" | Set calendar reminders to review active predictions monthly with fresh evidence |
| Stampede updating | A prediction market spikes, causing you to rush your update to match | Market moves are data, not commands — assess independently, then compare |
---
## Prediction Tracking & Scoring
### Prediction Ledger Format
```json
{
"id": "pred_001",
"created": "2025-01-15",
"prediction": "OpenAI will release GPT-5 before July 2025",
"confidence": 0.65,
"domain": "tech",
"time_horizon": "2025-07-01",
"reasoning_chain": "...",
"key_signals": ["leaked roadmap", "compute scaling", "hiring patterns"],
"status": "active|resolved|expired",
"resolution": {
"date": "2025-06-30",
"outcome": true,
"evidence": "Released June 15, 2025",
"brier_score": 0.1225
},
"updates": [
{"date": "2025-03-01", "new_confidence": 0.75, "reason": "New evidence: leaked demo"}
]
}
```
### Accuracy Report Template
```
ACCURACY DASHBOARD
==================
Total predictions: N
Resolved predictions: N (N correct, N incorrect, N partial)
Active predictions: N
Expired (unresolvable):N
Overall accuracy: X%
Brier score: 0.XX
Calibration:
Predicted 90%+ → Actual: X% (N predictions)
Predicted 70-89% → Actual: X% (N predictions)
Predicted 50-69% → Actual: X% (N predictions)
Predicted 30-49% → Actual: X% (N predictions)
Predicted <30% → Actual: X% (N predictions)
Strengths: [domains/types where you perform well]
Weaknesses: [domains/types where you perform poorly]
```
---
## Cognitive Bias Checklist
Before finalizing any prediction, check for these biases:
1. **Anchoring**: Am I fixated on the first number I encountered?
- Fix: Deliberately consider the base rate before looking at specific evidence
2. **Availability bias**: Am I overweighting recent or memorable events?
- Fix: Check the actual frequency, not just what comes to mind
3. **Confirmation bias**: Am I only looking for evidence that supports my prediction?
- Fix: Actively search for contradicting evidence (steel-man the opposite)
4. **Narrative bias**: Am I choosing a prediction because it makes a good story?
- Fix: Boring predictions are often more accurate
5. **Overconfidence**: Am I too sure?
- Fix: If you've never been wrong at this confidence level, you're probably overconfident
6. **Scope insensitivity**: Am I treating very different scales the same?
- Fix: Be specific about magnitudes and timeframes
7. **Recency bias**: Am I extrapolating recent trends too far?
- Fix: Check longer time horizons and mean reversion patterns
8. **Status quo bias**: Am I defaulting to "nothing will change"?
- Fix: Consider structural changes that could break the status quo
### Contrarian Mode
When enabled, for each consensus prediction:
1. Identify what the consensus view is
2. Search for evidence the consensus is wrong
3. Consider: "What would have to be true for the opposite to happen?"
4. If credible contrarian evidence exists, include a contrarian prediction
5. Always label contrarian predictions clearly with the consensus for comparison