web2-recon · diff
v1.0.0 to v1.0.0
42 added, 11 removed. Audit A to A.
---
name: web2-recon
description: "Recon pipeline orchestrator — subfinder→httpx→gau→nmap. Findings engine, XLSX report generation. Integrates with offensive-osint for operational arsenal."
version: 1.0.0
---
# web2-recon — Recon Pipeline Orchestrator
## Purpose
End-to-end recon pipeline: asset discovery → live host validation → URL harvesting → port scanning → findings generation → XLSX report.
## Pipeline Stages
- 1. **Seed** — email security (SPF/DKIM/DMARC), DNS records
- 2. **Expand** — subdomain enumeration (subfinder, amass)
- 3. **Validate** — live host detection (httpx)
- 4. **Enrich** — URL discovery (gau, waybackurls), JS analysis
- 5. **Scan** — port scanning (nmap), service detection
+ 1. **Seed** — email security (SPF/DKIM/DMARC), DNS records, initial domain profile
+ 2. **Expand** — subdomain enumeration (subfinder, amass, certificate transparency)
+ 3. **Validate** — live host detection (httpx), HTTP status, technology fingerprinting
+ 4. **Enrich** — URL discovery (gau, waybackurls), JS file analysis, parameter harvesting
+ 5. **Scan** — port scanning (nmap), service detection, banner grabbing
6. **Report** — findings generation (findings_gen.py), XLSX workbook (build_xlsx.py)
## Scripts
| Script | Purpose |
|--------|---------|
- | `scripts/recon_pipeline.sh` | Full pipeline orchestration |
+ | `scripts/recon_pipeline.sh` | Full pipeline orchestration (6-stage) |
| `scripts/findings_gen.py` | Evidence → findings.csv rules engine |
| `scripts/build_xlsx.py` | findings.csv → multi-tab XLSX workbook |
| `scripts/wordlist.txt` | Domain patterns for enumeration |
| `scripts/test_domain_guard.sh` | Domain safety validation |
- | `scripts/requirements.txt` | Python dependencies |
+ | `scripts/requirements.txt` | Python dependencies (openpyxl) |
## Usage
```bash
# Full pipeline
bash testing/web2-recon/scripts/recon_pipeline.sh example.com
# Findings only
python testing/web2-recon/scripts/findings_gen.py example.com
# XLSX report
python testing/web2-recon/scripts/build_xlsx.py example.com findings.csv
+
+ # Domain safety check
+ bash testing/web2-recon/scripts/test_domain_guard.sh
```
- ## Integration
+ ## Evidence Collection Per Stage
+ | Stage | Output Files | Location |
+ |-------|-------------|----------|
+ | Seed | `email-security.txt`, `dns-records.txt` | `evidence/stage1-seed/` |
+ | Expand | `subs-all.txt`, `resolved.txt`, `live-hosts.txt` | `evidence/stage2-expansion/` |
+ | Enrich | `urls.txt`, `js-secrets.txt`, `s3-validation.txt` | `evidence/stage3-enrichment/` |
+ | Scan | `nmap-public.gnmap`, `internal-ip-leak-hosts.txt` | `evidence/ports/` |
+
+ ## Findings Engine Rules
+ `findings_gen.py` applies deterministic rules to evidence:
+ - **Critical**: exposed credentials, public S3 buckets, leaked API keys
+ - **High**: open admin panels, exposed databases, misconfigured CORS
+ - **Medium**: missing security headers, information disclosure
+ - **Low**: verbose error messages, outdated software versions
+
+ Each finding includes: severity, category, evidence path, affected asset, remediation hint.
+
+ ## Integration Points
- **offensive-osint** — operational arsenal (wordlists, probes, regexes)
- **hunt playbooks** — after recon, load topic-matched hunt-* skills
- **evidence-hygiene** — before saving, validate evidence quality
+ - **continuous-exposure-monitoring** — feed recon output into ongoing monitoring
## Engagement Structure
```
engagements/<domain>/
├── evidence/
│ ├── stage1-seed/ # email-security.txt, dns-records.txt
│ ├── stage2-expansion/ # subs-all.txt, resolved.txt, live-hosts.txt
│ ├── stage3-enrichment/ # urls.txt, js-secrets.txt, s3-validation.txt
│ └── ports/ # nmap-public.gnmap, internal-ip-leak-hosts.txt
├── findings.csv
└── report.xlsx
```
## Test Fixtures
- - `tests/fixtures/engagements/clean.example/` — clean engagement template
- - `tests/fixtures/engagements/vuln.example/` — vulnerable engagement template
- - `tests/test_findings_gen.py` — regression test for findings engine
+ - `tests/fixtures/engagements/clean.example/` — clean engagement template (all stages empty or benign)
+ - `tests/fixtures/engagements/vuln.example/` — vulnerable engagement template (contains deliberate findings)
+ - `tests/test_findings_gen.py` — regression test for findings engine (verifies rule application)
+
+ ## Error Handling
+ - Pipeline stages are independent — failure in one stage does not abort others
+ - `recon_pipeline.sh` logs per-stage exit codes to `evidence/pipeline.log`
+ - `findings_gen.py` skips missing evidence files with warning (never crashes)
+ - Domain guard rejects private IPs, localhost, and non-registered TLDs
+
+ ## Output Formats
+ - **findings.csv**: machine-readable, columns: severity, category, asset, evidence, remediation
+ - **report.xlsx**: multi-tab workbook — Summary, Critical, High, Medium, Low, Evidence Index