--- name: bulk_refresh description: Run registered insight workflows over selected startup and community datasets with prerequisite ordering. Use for an intentional batch refresh or an externally scheduled run. --- # Bulk refresh Update dataset refresh states, then prepare selected datasets and execute their stage-mandatory or explicitly selected registry workflows. ## Operations and effects The async `bulk_refresh(datasets=None, skills=None, exclude=None)` returns `None` on success. Selectors are comma-separated strings or `all`. No dataset selector means active startup/community datasets after state updates; when skills are also omitted, include open, unarchived Application dossiers for screening. With explicit skills and no dataset selector, keep the active-dataset selection; `all` includes inactive ones. Named datasets may also be inactive and explicit selection does not activate them. Generated datasets are excluded. Omitted skills select mandatory skills per dataset's stage; `skills=all` selects the full applicable registry. Named skills expand their prerequisites. Domain restrictions always apply. ## Dataset scope and state First determine candidate datasets, resolve their Dealum stages and update markers using the existing local timestamps. Then acquire Dealum sources for active dossiers and selected open, unarchived Application dossiers, including newly discovered dossiers. Application dossiers are created without an active marker. All candidate states are decided before downloads begin; downloads cannot keep an already expired dossier alive. Non-Dealum dossiers use local files from the separate external sync. This step performs no parsing, embedding or insight generation. Named selectors limit all state updates and imports to those named datasets. A named missing startup is created only when its resolved Dealum stage is always active or Application; an absent match or other stage is an error without dossier creation. Startup aliases use the shared canonical slug. Broad selectors also discover missing dossiers in the always-active stages and, for default skill selection or an explicit dataset scope, Application, using the shared Dealum matcher and `ensure_startup_dossier`; historical pitched/rejected applications are not imported just because no local dossier exists. Existing archived dossiers in scope are still assigned their current stage, but are not downloaded or automatically reactivated. | Dealum stage | Refresh state | |---|---| | `Under Review`, `Jury`, `Pitching`, `Jury reserves (for pitching)` | Active regardless of age | | `Application`, `Rejected by Jury`, `Rejected for other reason`, `Not selected to pitch` | Three-month inactivity rule for existing dossiers | | `Pitched @ SICTIC` | Three-month inactivity rule | | Successfully confirmed absent from Dealum; community datasets | Three-month inactivity rule | The inactivity interval is three calendar months, configured in `config/bulk_refresh/lifecycle.json`. Use the latest file edit in `datasets/`, including the active marker and non-ingestible source files. Exclude hidden files, raw Dealum API JSON, manifests and the archived marker. Insights and parsed data are outside this source directory. A pitched stage observation never resets the active marker. Unchanged Dealum source files retain their timestamps on import. Active datasets outside the four always-active stages expire when their latest edit reaches the cutoff. An archived marker takes precedence over every stage: new source material and later stage changes never reactivate it. Only manual activation removes that block. An unmarked dataset defaults to archived unless an always-active stage applies. An unmarked dossier at stage `Application` stays unmarked, so it is activated once it reaches an always-active stage. Renaming the active marker to archived is not a qualifying source edit. Archiving changes eligibility only; it does not delete or zip anything. Application-stage screening does not activate unmarked dossiers. Existing active Application dossiers retain the inactivity rule; explicitly archived dossiers remain protected. Selected active Application dossiers are downloaded even with explicit skills. Explicit skills without a dataset selector do not discover new Application dossiers or include unmarked ones. Missing dossiers in Under Review, Jury, jury reserves and Pitching are still discovered normally. Pitched and rejected history is never imported merely because it exists in Dealum. Explicit dataset selection can process local archived files, but does not remove the archive or enable downloads. Missing credentials, failed lookups, ambiguous applications and unknown/missing stages are errors, not “absent from Dealum.” Affected startup work is skipped; unrelated datasets continue and the run raises at the end. Community datasets do not require a Dealum lookup. A broad discovery failure is also reported. ## Mandatory skills The existing registry declares `mandatory_stages` alongside domains and dependencies. `"*"` means every stage; `None` means confirmed absent from Dealum. Stage-specific mandatory declarations apply to startups, not community datasets; wildcard declarations also apply to supported community datasets. | Skill | Mandatory stages | |---|---| | `startup-profile`, `persons-in-dataset` | All supported datasets except Application-stage startups | | `submission-ready` | `Application`, `Under Review` | | `startup-website-import` | `Jury`, jury reserves, `Pitching`, `Pitched @ SICTIC` | | `startup-traction` | `Pitched @ SICTIC`, absent from Dealum | Application-stage defaults run only `submission-ready`. Registry `excluded_mandatory_stages` suppresses the wildcard profile/discovery defaults; explicit skill selections and dependencies remain unaffected. `person-profile` is not mandatory at any stage. It remains available through explicit skill selection, `--skills all`, or as a prerequisite of a selected skill. `jury-review` and company research are future skills, not executable placeholders. Add their stage declarations when implementing them. Mandatory means invoke with normal cache reuse, not force regeneration. Selected `prepares_sources` jobs finish before insight jobs for the same dataset; bulk execution synchronizes their acquired content. This is execution ordering, not a required dependency: failed website acquisition is logged and reported, while insights continue with available evidence. Only declared `depends_on` dependencies propagate skill-failure skips. Explicit skill lists do not implicitly add website crawling. ## Execution and compatibility `exclude` / `--exclude` accepts comma-separated source dataset names using the same normalization as `--datasets`. With exclusions and no dataset selector, start from all startup/community datasets, including inactive ones. Otherwise subtract exclusions from the explicit selection. Unknown excluded source names raise before preparation; known names outside the selection have no effect. Excluding every selected dataset logs and returns without doing work. After target resolution, select skills and build dataset–skill jobs and their dependencies. Then synchronize selected datasets before scheduling those jobs. Source acquisition during target resolution is separate from this expensive parsing/indexing step. The scheduler supports async callables and runs synchronous callables in worker threads, allowing the canonical website importer to register directly without another entry point. Independent ready jobs run concurrently. Cross-domain dependencies cover prerequisite nodes within the selected dataset scope; they do not add datasets. The [registry](../skill_registry.py) owns supported domains and dependencies. Pre-ingestion failures skip affected jobs and their dependants. Skill failures also propagate skips, while independent work continues. Completed artifacts remain saved. After runnable work finishes, problems raise `BulkRefreshError`; an empty dataset selection logs and returns. Side effects are those of preparation and selected skills, including source imports, discovery, enrichment, conversion, indexing and report generation. Each skill owns artifact reuse. The command does not itself install a schedule. One run shares its Dealum application snapshot and successful imports across composed workflows. This run-local reuse does not change standalone import calls. Google Drive synchronization and cron scheduling remain external. ## Usage ```bash conda run -n sictic-env python -m skills.bulk_refresh --datasets example-startup,sictic-members --skills expert-search conda run -n sictic-env python -m skills.bulk_refresh --datasets all --skills startup-profile conda run -n sictic-env python -m skills.bulk_refresh --exclude sictic-members --skills persons-in-dataset ``` This operational tool has no harness command. ## References - [Implementation](bulk_refresh.py), [CLI](__main__.py) - [Registry contract](../standards_and_architecture/SKILL.md#bulk-refresh-registry)