startup_website_import git:20261020.f81ab8bA

Immutable. This exact content never changes and is served at /api/v1/blob/d00728a815b6cf12.

---
name: startup_website_import
description: Import a startup website once as Markdown pages and linked documents, identifying formats from downloaded content. Accept a startup and optional explicit URL.
---

# Startup website import

Collect website source material for later dataset ingestion.

## Operations and effects

The synchronous `startup_website_import(startup_name, url=None, ...)` returns
`WebsiteImportResult` with paths and counts, not insight artifacts.
This is the single entry point for direct, CLI, discovery and bulk calls.
If `datasets/website/` already contains source files, skip without any network
requests or writes, even when a URL is explicitly supplied. There is no force
or re-scrape option. Skips return zero new-file counts and `skipped_reason`.
This check precedes option validation, configuration loading and URL resolution.
When absent, a shared filesystem lock serializes acquisition for the dossier;
the snapshot check is repeated after locking to prevent concurrent re-scraping.

When URL is omitted, use the existing `website_from_evidence` selector with
current raw Markdown and available parsed sources, including startup aliases.
Missing or ambiguous URLs are logged and skipped, not guessed. A first crawl
creates the dossier without activating it, then stages the snapshot before saving
`datasets/website/`.

The crawler queues same-host links, excludes configured low-value path segments,
and defaults to depth 1 and 50 HTML pages. Document downloads default to enabled,
with 20 documents and 25 MB per response. Robots rules are used when available; failure to load them
does not stop the crawl. Requests follow HTTP redirects, but destinations are
validated before any content or links are saved.
Defaults live in `config/startup_website_import/crawl.json` and are shared by
bulk refresh, person discovery and the CLI. Explicit crawl options override
those defaults only for a first acquisition; they never force a new crawl.

### Website relevance checks

`config/startup_website_import/content_policy.json` owns the deterministic rules:

- Amazon marketplace domains and short links are not company websites. The shared
  `website_from_evidence` selector ignores them; explicit inputs are logged and
  skipped before fetching. Book/product links in original source documents remain
  intact, but cannot initiate an Amazon storefront crawl.
- HTTP/HTTPS and `www` hostname changes are accepted. Other hostname changes are
  skipped unless explicitly verified in `approved_redirect_hosts`, a mapping from
  source hostname to a list of permitted destination hostnames (lowercase, without
  `www`). This applies to subpages and document downloads as well as the homepage.
  Alternatively, supply the verified new website URL explicitly. Company-name
  text on a destination page alone does not establish ownership. Rejected
  destinations are neither saved nor followed; an accepted homepage hostname
  change reloads robots rules for that host.
- HTML titles, main headings, visible text and link targets are screened for
  configured gambling-spam signatures. Gambling landing pages are rejected.
  Short isolated spam labels/links are removed before Markdown rendering, link
  discovery and crawl expansion. Pages dominated by spam, or with longer spam
  blocks that cannot safely be isolated, are skipped. Ordinary mentions of casinos
  or gambling in legal terms, business descriptions or articles are retained.

These checks use no LLM calls. They log rejection/cleanup reasons and preserve the
existing failed-page count and zero-HTML failure behavior. They do not prove company
identity or detect every form of spam. Existing snapshots are still reused first;
the importer does not automatically rewrite or re-index old source material.

Every fetched response is classified by the shared
`lib.infrastructure.document_formats.detect_document_extension` before saving.
Content takes precedence over URL suffixes and HTTP Content-Type. HTML becomes
Markdown through the existing page renderer; supported documents are saved with
their detected extension. Supported downloads are PDF, DOCX, XLS/XLSX/XLSM, PPTX,
RTF, TXT, Markdown, JSON and CSV. Office ZIP containers are inspected for their
document structure; generic archives, images, other unsupported formats and
unidentified binary content are logged and skipped. Format detection does not
guarantee that a document is uncorrupted; full validation remains with conversion.
Text uses UTF-8, BOM-marked UTF-16 or an explicitly declared HTTP/HTML charset;
saved text is normalized to UTF-8. Undecodable content is skipped.
Markdown/CSV names or headers only disambiguate verified text, never binary files.

Fetch each requested URL once. Resolve page links relative to the final URL after
redirects and deduplicate saved responses by that URL. A document link redirected
to an already saved homepage creates no duplicate and no false document.
Known document links can be acquired from pages at the crawl-depth limit;
extensionless links follow normal page-depth limits and are classified on fetch.

Save Markdown pages with source metadata, PDFs under `website/pdfs/`, other
documents under `website/documents/`, plus
`linkedin-urls.md` and `linkedin-and-resume-links.md`. These are discovered
links and possible-resume hints, not verified person identities.

Failed or unsupported responses are logged and skipped and counted in the existing
`failed_pages` field. `documents_saved` counts all saved documents, including PDFs;
`pdfs_saved` retains its PDF-only meaning. Zero saved HTML pages raises `InfrastructureError` with provider `website`,
operation `import` and kind `SERVICE_UNAVAILABLE` (still a `RuntimeError`), and leaves
other source data intact and remains retryable. Once a crawl succeeds, its snapshot
is retained for future calls. A sibling staging directory is published through
the shared atomic directory-snapshot helper, with rollback on rename failure.
A failed publication leaves no partial new snapshot that would block retries.
Other dossier data is retained. The direct crawler performs no indexing, profile
generation or LinkedIn enrichment, and has no harness registration.

## Bulk execution

The registry's `startup-website-import` entry points directly to the same function.
Bulk refresh executes synchronous callables in a worker thread. The registry marks
this as source preparation: bulk refresh synchronizes after acquisition, before
running dependent insight jobs. No separate bulk wrapper or scraping policy exists.
Partial crawls are logged; technical failures propagate through bulk refresh.

## Usage

```bash
conda run -n sictic-env python -m skills.startup_website_import example https://example.org --depth 1
conda run -n sictic-env python -m skills.startup_website_import example
```

The direct CLI takes positional `STARTUP` and optional `URL` arguments and exposes
`--depth`, `--max-pages`, `--documents/--no-documents`,
`--max-documents`, `--max-document-mb` and `--respect-robots/--ignore-robots`.
The former `--pdfs/--no-pdfs`, `--max-pdfs` and `--max-pdf-mb` options remain
aliases for document controls. Python accepts `include_documents`, `max_documents`
and `max_document_mb`, with the former PDF arguments retained as aliases; explicitly
supplied document arguments take precedence. Configuration retains its existing PDF
key names for compatibility, now applying to the combined document downloads.
Existing snapshots are still reused without reclassification or automatic repair.

## References

- [Implementation](startup_website_import.py), [CLI](__main__.py)
- [Dataset paths and storage](../standards_and_architecture/SKILL.md#datasets-and-storage)