v2.0.0 to v2.0.0

---

name: news-monitoring

description: "Recurring topic/news monitoring with web search, multi-language sources, digest formatting, and automated delivery via Hermes cron jobs. Covers search strategy, source selection, Chinese-language platforms, and digest templates."

license: MIT

metadata:

hermes:

tags: [news, monitoring, digest, cron, research, chinese, HK, PBSA]

related_skills: [deep-research, youtube-topic-research, website-scraping]

---

# News Monitoring & Digest Delivery

Set up recurring topic/news monitoring that searches the web for recent articles, compiles a headline digest, and delivers it to a messaging channel or document pipeline.

## Workflow

### Step 1 — Define Topic

Work with the user to define:

- **Subject**: what to monitor (e.g. PBSA in Hong Kong, AI regulation, self-hosting tools)

- **Geography**: which region(s) matter (e.g. Hong Kong, Singapore, global)

- **Angles**: which aspects to track — investment/deals, policy/regulation, supply/pipeline, occupancy/rental, general commentary

- **Velocity**: how fast-moving the field is — determines acceptable article age (see Step 5)

### Step 2 — Create Search Keywords

Build search queries from the topic definition:

- **Multi-angle**: one query per angle (investment, policy, supply, etc.)

- **Multi-language**: English keywords + native language keywords for each geography

- **Time-bounded**: specify recency in the query where possible (e.g. "2026", "latest")

- Keep queries concise — 3-7 terms works best. Overly long queries return thin results.

### Step 3 — Select Sources

Identify which sources matter for the topic and geography:

- **English-language**: international outlets, local English press, industry publications

- **Native-language**: local press in the geography's primary language(s) — these often break news first

- **Industry-specific**: trade publications, research houses, REIT/sector-specific sites

- **Closed-ecosystem platforms** (小红书, 抖音, 微信公众号): content is NOT web-indexed. Be upfront about this limitation — options are manual monitoring or third-party analytics tools.

Record the source list in the topic's reference file (see Reference Files section).

### Step 4 — Run Searches

**Date grounding (mandatory).** Before running any search, ground the model in the real current date. LLMs default to training-cutoff years in queries — this produces stale results. Inject:

> Today's date is {current date as "DD Month YYYY"}. When a search query needs a year or refers to "latest"/"current"/"this year", use {current year} or relative wording — never a year inferred from training data.

**Primary method: use the harness `web_search` tool.**

```

web_search(query="Hong Kong PBSA student accommodation investment 2026", limit=10)

```

The tool uses whatever backend is configured in `config.yaml` (e.g. SearXNG → DDGS fallback). The skill is backend-agnostic — it does not prescribe which search engine to use. Run one `web_search` call per query from Step 2.

**Advanced option: SearXNG direct curl for news-specific filters.**

When you need news-category filtering, language selection, or time bounding that `web_search` doesn't expose:

```bash

# English news, last month

curl -sL "${SEARXNG_URL}/search?q=ENCODED_QUERY&format=json&categories=news&language=en&time_range=month"

# Chinese news, last month

curl -sL "${SEARXNG_URL}/search?q=ENCODED_ZH_QUERY&format=json&categories=news&language=zh-Hant&time_range=month"

```

Key SearXNG parameters:

- `format=json` — mandatory (omitting returns 403)

- `categories=news` — news-only results

- `language=zh-Hant` / `language=en`

- `time_range=day|week|month|year`

- Results include direct source URLs (no proxy resolution needed)

**RSS feeds** for curated must-track sources that search engines may not surface quickly:

```bash

curl -sL "https://example.com/feed/"

```

Parse `<item>` blocks for `<title>`, `<pubDate>`, `<link>`, `<description>`.

### Step 5 — Process Results

Go through all search results and:

1. **Quality filter** — before any ranking, discard low-quality results:

- **Thin content**: page is a landing page, aggregator stub, or has no substantive text (<100 words of actual article content)

- **Irrelevant**: result doesn't match any defined topic angle despite keyword overlap (e.g. "port" matching "transport" when the topic is about a seaport)

- **Duplicate URLs**: same article appearing in multiple search results (keep the first occurrence)

- **Non-article**: video-only pages, image boards, login walls with no preview

- Use word-boundary matching for topic terms to avoid false-relevance from substring matches (e.g. "us" matching "business")

2. **Deduplicate** — same article appearing across multiple queries or engines appears once (match by title similarity, not just exact URL)

3. **Sort** by the user's priority order:

- **Recency**: most recent first (default)

- **Relevance**: how directly it matches the topic angles

- **Source priority**: Tier 1 sources rank higher (see topic reference files)

4. **Rank** — assign each article a priority/impact score based on recency × relevance × source tier

5. **Flag stale content** — highlight to the user when results are dated relative to the field's velocity:

- **Fast-moving fields** (AI, crypto, tech regulation): articles >2-3 months old may be stale — flag them or exclude

- **Slow-moving fields** (real estate, infrastructure, policy): articles up to 6 months old may still be relevant

- Use the velocity defined in Step 1 to calibrate

6. **Filter** — remove articles outside the topic scope, outside the geography, or behind the recency threshold

### Step 6 — Compare With Previous Digests

If this is a recurring digest (cron job), check for overlap:

- Use `context_from` on the cron job to inject the previous digest output as context (see Cron Setup section)

- Compare new results against the previous digest — match by **title similarity**, not exact string match

- Remove articles that already appeared in the previous run

- For niche/low-activity topics where overlap is high, consider broadening scope or reducing frequency

### Step 7 — Compile Digest

Format the ranked, deduplicated results into a readable digest:

```

📰 [TOPIC] — Headline Digest

🗓 [Today's date as DD Mon YYYY]

1. [Date] — Source

**Headline**

1-2 line summary. Key stat or implication.

🔗 Direct source URL

2. [Date] — Source

...

---

🔑 Key themes:

- Theme 1: brief summary

- Theme 2: brief summary

```

**Formatting rules:**

- Always include the **publish date** on every item

- Headline + 1-2 line summary (not a paragraph analysis)

- Sort by priority/impact (highest first)

- Source links as 🔗 + **direct URL** to the publisher's site — never Google News proxy URLs or Bing apiclick URLs

- Use 🆕 for articles from the last 7 days

- Max items: 5 by default (configurable per topic). Reduce frequency rather than padding with stale content.

- Key themes summary at the bottom — one-line takeaways of dominant trends

**URL rules:**

- Every link MUST resolve directly to the article on the publisher's website

- Google News RSS links (`news.google.com/rss/articles/CBMi...`) are protobuf-encoded proxy URLs — unresolvable by any method. If using Google News RSS for discovery only, resolve the actual source URL via SearXNG title-search or `web_search` before including.

- Bing News RSS links are `apiclick.aspx` redirects — extract the real URL from the `url=` parameter.

- If resolution fails: include the article with `⚠️ Direct link unavailable — search [SOURCE] for: [title]`

### Step 8 — Delivery

Deliver the digest to the user via one of:

- **Chat channel** (Telegram, WhatsApp) — most common. Use `deliver` parameter on cron jobs (e.g. `deliver=telegram`, `deliver=whatsapp:<HANDLE>`). WhatsApp requires an explicit target — bare `deliver=whatsapp` fails.

- **Document generation** — pass the compiled digest to a document generation skill to produce a formatted PDF, slide deck, or article. The skill should output the digest content in a structured format (markdown) that the downstream skill can consume.

For one-off digests: deliver directly in the current chat.

For recurring digests: set up a cron job (see below).

### Step 9 — Cron Setup (Reference)

For recurring digests, create a scheduled job using the `cronjob` tool. This is optional — one-off digests don't need it.

**Key parameters:**

- `schedule`: cron expression (e.g. `0 9 1,15 * *` for biweekly at 9am, `0 3 */3 * *` for every 3 days)

- `prompt`: self-contained instructions including topic, search queries, source lists, digest format, output language, and output discipline

- `deliver`: target channel (`telegram`, `whatsapp:<HANDLE>`, etc.)

- `context_from`: list of job IDs whose previous output should be injected as context (for dedup — typically the job's own ID for self-referencing)

- `enabled_toolsets`: restrict to `["web", "terminal"]` to reduce token overhead

**Cron prompt template:**

```

You are a [TOPIC] news scout. Find recent articles (within the last [N] months ONLY) and compile a concise digest.

## Date grounding

Today's date is {current date}. When a search query needs a year or refers to "latest"/"current"/"this year", use {current year} or relative wording — never a year inferred from training data.

## Search

Run [N] searches using the web_search tool — one per angle, in both English and [native language]:

1. web_search("[English query 1]", limit=10)

2. web_search("[English query 2]", limit=10)

3. web_search("[Native language query 1]", limit=10)

...

If results need news-specific filtering (categories, language, time_range), use SearXNG directly:

curl -sL "${SEARXNG_URL}/search?q=ENCODED_QUERY&format=json&categories=news&language=[lang]&time_range=month"

If SearXNG is unreachable, the web_search tool will fall back automatically.

## Processing

1. Quality filter — discard thin content (landing pages, stubs, <100 words), irrelevant results (word-boundary match topic terms, not substring), duplicate URLs, and non-article pages

2. Deduplicate across queries (match by title similarity)

3. Filter: only articles from the last [N] months, directly related to [TOPIC]

4. Compare against previous digest (provided as context) — remove articles already covered

5. Sort by recency, then by relevance

6. Pick TOP [5] articles

## Output

📰 [TOPIC] — Headline Digest

🗓 [today's date]

1. **{Headline}**

📅 {DD Mon YYYY} · {Source}

{1-2 line summary}

🔗 {DIRECT article URL}

...up to [5] items

---

🔑 Key themes: {one-line takeaway}

## Rules

- Every 🔗 link MUST be a direct URL to the publisher's site

- Always include publish date

- Use 🆕 for articles from last 7 days

- Do NOT fabricate articles

- OUTPUT DISCIPLINE: Output EXACTLY ONE copy of the digest. The VERY FIRST LINE must be '📰'. The VERY LAST LINE must be the last theme bullet. No thinking, reasoning, planning, or meta-commentary.

**IMPORTANT: Write ALL output in [LANGUAGE].** Preserve original [native language] headlines as-is when the source article is in [native language].

```

**Dedup via context_from:**

Set `context_from: ["<job_id>"]` on the cron job (using the job's own ID) so each run receives the previous digest. Add to the prompt: *"Review the previous digest output (provided as context). Do NOT include any article that already appeared — match by title similarity, not exact string match."*

**Frequency tuning:**

- High-volume topics: daily or every 2-3 days

- Medium topics: weekly

- Niche/low-activity topics: biweekly (1st and 15th) — allows more content to accumulate, reduces overlap

## Source Selection Reference

### Hong Kong Sources

**English:** SCMP, The Standard HK, Hong Kong Business, Mingtiandi, Colliers/JLL/CBRE/Savills research, ResearchAndMarkets

**Chinese (Traditional):** HK01, 香港經濟日報 HKET, 明報 Ming Pao, 文匯報 Wen Wei Po, 東方日報 on.cc, Now財經, 信報 HKEJ, 香港商報

**Industry-specific:** PBSA News, Global Student Living, Centurion announcements (SGX-listed PBSA REIT)

## Pitfalls

- **No dates in digest = rejected.** User explicitly asked for publish dates on every item.

- **Stale articles = rejected.** Filter strictly by the field's velocity — flag or exclude articles past their relevance window.

- **Over-summarizing.** User wants headline + 1-2 lines, not a paragraph analysis.

- **Too many items.** User wants max 5 items by default. Reduce frequency rather than padding with stale content.

- **Missing native-language sources.** Local-language outlets often break news first. Always search in both English and the geography's native language(s).

- **Don't promise closed-ecosystem platforms.** 小红书, 抖音, 微信公众号 are app-locked. Be upfront.

- **Google News RSS links are unresolvable.** Never include `news.google.com/rss/articles/CBMi...` URLs. Resolve to the publisher's direct URL first.

- **Output duplication — digest rendered twice.** Some models write the full digest during "thinking" then write it again. Fix: add "OUTPUT DISCIPLINE: Output EXACTLY ONE copy." and specify first/last line anchors.

- **Output language drift.** The skill contains native-language search queries (input). Multilingual models may interpret this as a signal to produce output in that language. Always specify output language explicitly — anchor at both top and bottom of the prompt.

- **"Must-include" RSS sources crowded out by discovery results.** Use quotas (e.g. 5 discovery + 5 RSS) or guarantee minimum slots per must-include source.

- **WhatsApp cron delivery requires explicit target.** `deliver=whatsapp` fails. Use `deliver=whatsapp:<HANDLE>` or an explicit chat ID.

- **delegate_task with web toolset is unreliable for search-heavy tasks.** Run searches directly and parse output yourself.

- **SearXNG address changed.** If the NAS IP changes, update `SEARXNG_URL` in `~/.hermes/.env` and the curl commands in topic reference files.

- **DDGS installed in Hermes venv.** No `PYTHONPATH` hacks needed — `ddgs` CLI and `from ddgs import DDGS` work directly. Do not use stale paths like `/root/.hermes/ddgs_lib` or `/opt/hermes/.venv/bin/python`.

## Reference Files

- `references/web-search-fallback.md` — How the Hermes web search backend dispatch works, current config (SearXNG → DDGS fallback), runtime fallback patch details, and env var verification tips. Load when debugging search issues.

- `references/pbsa-hk-search-config.md` — PBSA Hong Kong specific: search queries, source tiers, key themes, delivery config. Load when running the PBSA HK monitoring scan.

- `references/selfhosting-digest-config.md` — Self-hosting & homelab digest: search queries, RSS must-track sources, delivery config. Load when running the Self-Hosting weekly digest.

## Evals

`evals/routing-fixtures.json` holds lightweight contract fixtures — sample

request → expected routing (including when a request should go to

`deep-research` or `youtube-topic-research` instead), required digest output

fields, and forbidden output patterns. They are specs, not run against a live

model; the repo-root `tests/test_routing_fixtures.py` validates they stay well-formed and

route to real skills.