toolforge-nodejs · git:20260701.852b384 · 2026-07-01 · sha256 c98b6560b392d147
toolforge-nodejs git:20260701.852b384A
Immutable. This exact content never changes and is served at /api/v1/blob/c98b6560b392d147.
---
name: toolforge-nodejs
description: Deploy and manage Node.js web services on Wikimedia Toolforge Kubernetes — zero-dependency server patterns, webservice commands, PORT configuration, static file serving with caching headers, npm on NFS, environment variables, logging, and common pitfalls
license: MIT
compatibility: opencode
depends_on: [wikimedia-toolforge, wikimedia-api-access]
skill_discovery_hints:
- keywords: ["Toolforge Node", "Node.js", "Node webservice", "node22", "node18", "npm", "package.json"]
- keywords: ["Toolforge port", "PORT env", "webservice port", "Toolforge Kubernetes", "web service"]
- keywords: ["Toolforge static files", "express", "http server", "zero dependency", "built-in http"]
- keywords: ["Cache-Control", "Varnish", "Toolforge proxy", "immutable cache", "static asset"]
- keywords: ["Toolforge environment", "env vars", "secrets", "toolforge env", "config", "environment"]
- keywords: ["Toolforge logs", "kubectl logs", "webservice logs", "debug", "tail logs", "toolforge debug"]
- keywords: ["Node streaming", "buffer", "readFile", "createReadStream", "Pannellum", "XHR", "WebGL"]
- keywords: ["Toolforge cron", "node job", "scheduled task", "cron node", "batch job"]
last_verified: 2026-06-20
---
> ⚠️ **Prerequisites:** This skill assumes you have a Toolforge account and can SSH in. See **[wikimedia-toolforge](../wikimedia-toolforge/SKILL.md)** for account setup, tool creation, and basic SSH configuration.
> 💡 **Zero dependencies by default:** Toolforge's NFS filesystem has slow metadata operations — `npm install` can take minutes. The patterns in this skill use **only Node.js built-in modules** by default, which means zero `npm install`, zero `package.json`, and instant deployment. Add dependencies only when you must.
> ⚠️ **Toolforge Rule #2 — Open Source License Required:** All code in the Tools
> project must be published under an [OSI-approved](https://opensource.org/licenses)
> open source license. Add a `LICENSE` file to your repository before deploying.
> The absence of a license means default copyright laws apply, which is counter to
> the principles of the Wikimedia movement. See the
> [full rules](https://wikitech.wikimedia.org/wiki/Help:Toolforge/Rules).
---
## Quick Reference
| Goal | Command / Pattern |
|------|-------------------|
| Deploy a new Node.js service | See SOP 1 |
| Choose a runtime | `node22` (default), `node18` (legacy) — see SOP 1.1 |
| Start/restart a webservice | `become <tool> webservice --backend=kubernetes node22 start` |
| See logs | `become <tool> kubectl logs -f deployment/<tool>` |
| Set environment variables | `become <tool> toolforge env set NAME value` |
| Serve static files | Use `readFile()` with correct MIME types — see SOP 3 |
| Add npm dependencies | Use a `package.json` and `npm install` on the tool's NFS — see SOP 5 |
| Run a cron job | `become <tool> job schedule node22 --command 'node script.mjs' "0 * * * *"` |
---
## SOP 1: Bootstrap a Zero-Dependency Node.js Server
The fastest way to get a Node.js web service running on Toolforge — **no `npm install`, no `package.json`**, just a single `.mjs` file.
### 1.1 Choose a Runtime
Toolforge offers these Node.js runtime versions:
| Runtime | Kubernetes Image | Status |
|---------|-----------------|--------|
| `node22` | `docker-registry.tools.wmflabs.org/toolforge-node22-sssd-web:latest` | ✅ Current default (Node 22.x) |
| `node18` | `docker-registry.tools.wmflabs.org/toolforge-node18-sssd-web:latest` | ⚠️ Legacy (Node 18.x, available but not recommended for new tools) |
The runtime image includes the operating system, Node.js binary, and basic tools. Your code lives on NFS at `/data/project/<tool>/`.
### 1.2 Create the Tool (One Time)
```bash
# From your local machine
ssh <user>@login.toolforge.org toolforge tools create my-tool
```
### 1.3 Write Your Server
Create a single-file server that uses only Node.js built-in modules. The key detail: **Toolforge sets the `PORT` environment variable** — your server must bind to it.
```javascript
// server.mjs — Zero-dependency Node.js web service for Toolforge
import { createServer } from 'node:http';
import { readFile } from 'node:fs/promises';
import { existsSync } from 'node:fs';
import { join, extname, resolve as pathResolve, sep as pathSep } from 'node:path';
import { fileURLToPath } from 'node:url';
const __dirname = fileURLToPath(new URL('.', import.meta.url));
const PORT = process.env.PORT || 8765;
// MIME types for static files
const MIME = {
'.html': 'text/html; charset=utf-8',
'.css': 'text/css; charset=utf-8',
'.js': 'application/javascript; charset=utf-8',
'.mjs': 'application/javascript; charset=utf-8',
'.json': 'application/json; charset=utf-8',
'.png': 'image/png',
'.jpg': 'image/jpeg',
'.jpeg': 'image/jpeg',
'.svg': 'image/svg+xml',
};
const server = createServer(async (req, res) => {
const url = new URL(req.url, `http://localhost:${PORT}`);
const pathname = url.pathname;
// API routes
if (pathname === '/api/hello' && req.method === 'GET') {
res.writeHead(200, { 'Content-Type': 'application/json' });
res.end(JSON.stringify({ message: 'Hello from Toolforge!' }));
return;
}
// Static files
try {
const filePath = join(__dirname, 'public', pathname === '/' ? 'index.html' : pathname);
// Security: prevent directory traversal — resolve and verify containment
// startsWith without trailing sep matches sibling dirs (e.g. my-tool2)
const safeDir = __dirname + path.sep;
const resolved = path.resolve(filePath);
if (!resolved.startsWith(safeDir)) {
res.writeHead(403); res.end('Forbidden');
return;
}
const data = await readFile(filePath);
const ext = extname(filePath);
res.writeHead(200, {
'Content-Type': MIME[ext] || 'application/octet-stream',
'Content-Length': data.length,
// One-week browser cache for static assets
'Cache-Control': pathname.startsWith('/images/')
? 'public, max-age=604800, immutable'
: 'public, max-age=3600',
});
res.end(data);
} catch {
res.writeHead(404, { 'Cache-Control': 'no-store' });
res.end('Not found');
}
});
server.listen(PORT, () => {
console.log(`Server running on port ${PORT}`);
});
```
**Naming convention:** Use `.mjs` extension for ES module syntax. If you need `require()`, use `.cjs`. For mixed use, a `package.json` with `"type": "module"` makes `.js` files ES modules.
### 1.4 Deploy
```bash
# From your local machine
scp server.mjs <user>@login.toolforge.org:/data/project/my-tool/
# Create a public/ directory for static files
ssh <user>@login.toolforge.org "mkdir -p /data/project/my-tool/public"
# Copy your static files
scp index.html style.css <user>@login.toolforge.org:/data/project/my-tool/public/
# Start the webservice
ssh <user>@login.toolforge.org "become my-tool webservice --backend=kubernetes node22 start"
```
### 1.5 Verify
```bash
# Check status
ssh <user>@login.toolforge.org "become my-tool webservice --backend=kubernetes node22 status"
# View logs
ssh <user>@login.toolforge.org "become my-tool kubectl logs -f deployment/my-tool"
# Test via curl or browser
curl https://my-tool.toolforge.org/
curl https://my-tool.toolforge.org/api/hello
```
---
## SOP 2: The `PORT` Environment Variable
Toolforge's Kubernetes proxy routes incoming requests to your container. It **ignores** any port you specify in code and expects your server to listen on the port given by the `PORT` environment variable.
### Do This
```javascript
const PORT = process.env.PORT || 8765; // 8765 is a safe local dev default
server.listen(PORT, () => console.log(`Listening on ${PORT}`));
```
### Don't Do This
```javascript
server.listen(3000); // ❌ Toolforge proxy won't find it
server.listen(80); // ❌ Not running as root
server.listen('8080'); // ❌ PORT is a number, not a string
```
### How It Works
```
Browser ─→ toolforge.org ─→ Kubernetes Ingress ─→ Pod:PORT
│
process.env.PORT
(e.g., "31245")
```
The `PORT` value is assigned per-pod and is **not predictable** — always read it from the environment.
---
## SOP 3: Static File Serving with Correct Caching
### Cache-Control Headers
Toolforge uses **Varnish** as an HTTP accelerator in front of Kubernetes. Setting proper cache headers is critical for performance:
| Asset Type | Recommendation | Effect |
|------------|---------------|--------|
| Images (panoramas, thumbnails) | `public, max-age=604800, immutable` | 1 week, never revalidates |
| JS/CSS (versioned) | `public, max-age=31536000, immutable` | 1 year, never revalidates |
| HTML pages | `public, max-age=3600` | 1 hour, revalidates |
| API responses | `no-cache` (or omit caching) | Always revalidates |
| Error pages (404, 500) | `no-store` | Never caches errors |
```javascript
function cacheHeaders(pathname) {
if (pathname.startsWith('/images/')) {
return 'public, max-age=604800, immutable'; // 1 week
}
if (pathname.match(/\.(js|css|mjs)$/)) {
return 'public, max-age=31536000, immutable'; // 1 year
}
if (pathname.startsWith('/api/')) {
return 'no-cache'; // API responses
}
return 'public, max-age=3600'; // HTML and everything else
}
```
### Why `immutable` Matters
The `immutable` directive tells the browser it never needs to revalidate the resource before its `max-age` expires. For panorama images that are cache-busted with `?v=` timestamps, this saves a round-trip on every page load. Without it, browsers send `If-Modified-Since`/`If-None-Match` headers even for cached assets.
### Never Cache Error Responses
```javascript
// ❌ Wrong — browser caches the 404 page forever
res.writeHead(404, { 'Cache-Control': 'public, max-age=604800' });
res.end('Not found');
// ✅ Correct — browser never caches error responses
res.writeHead(404, { 'Cache-Control': 'no-store' });
res.end('Not found');
```
### MIME Types Must Be Explicit
Node.js `http` module doesn't set MIME types automatically. Missing or wrong MIME types cause browsers to refuse to load resources:
```javascript
const MIME = {
'.html': 'text/html; charset=utf-8',
'.css': 'text/css; charset=utf-8',
'.js': 'application/javascript; charset=utf-8',
'.mjs': 'application/javascript; charset=utf-8',
'.json': 'application/json; charset=utf-8',
'.png': 'image/png',
'.jpg': 'image/jpeg',
'.jpeg': 'image/jpeg',
'.svg': 'image/svg+xml',
};
// Always set Content-Type and Content-Length
res.writeHead(200, {
'Content-Type': MIME[ext] || 'application/octet-stream',
'Content-Length': data.length,
});
```
---
## SOP 4: Serving Images for WebGL/Pannellum — Never Stream
Pannellum's 360° viewer loads panorama images via `XMLHttpRequest` and expects a **complete, buffered response**. Streaming responses (using `createReadStream().pipe(res)`) causes `net::ERR_FAILED 200 (OK)` — the browser sees HTTP 200 but can't read the response body as a Blob.
### ❌ Broken: Streaming
```javascript
// DO NOT USE — Pannellum fails with ERR_FAILED
if (stat.size > 1024 * 1024) {
const { createReadStream } = await import('node:fs');
createReadStream(filePath).pipe(res);
return;
}
```
### ✅ Correct: Buffering
```javascript
// Always read the entire file into memory before responding
const data = await readFile(filePath);
res.writeHead(200, {
'Content-Type': 'image/jpeg',
'Content-Length': data.length,
'Cache-Control': 'public, max-age=604800, immutable',
});
res.end(data);
```
**Why buffering is safe on Toolforge:** Your images are on the local NFS mount — reads are fast. The memory cost (a few MB per image) is negligible on Toolforge's Kubernetes pods with 2GB+ RAM limits.
### When Streaming IS Safe
| Use Case | Method | Why |
|----------|--------|-----|
| `<img>` tags | Streaming works | Browser handles streamed images natively |
| `fetch()` + `response.blob()` | Streaming works | Different internal path than XHR |
| Audio/video file serving | Streaming preferred | Media elements handle buffered seeking |
| Large file downloads | Streaming preferred | Lower memory footprint |
---
## SOP 5: Adding npm Dependencies
For most Wikimedia tools, the built-in `http` module is sufficient. When you need external packages:
### 5.1 Create `package.json`
```json
{
"name": "my-tool",
"version": "1.0.0",
"type": "module",
"dependencies": {
"express": "^4.18.0"
},
"scripts": {
"start": "node server.mjs",
"dev": "node --watch server.mjs"
}
}
```
The `"type": "module"` field makes all `.js` files use ES module syntax (`import`/`export`). Use `.cjs` extension for files that need CommonJS (`require`).
### 5.2 Install on Toolforge
NFS metadata operations are slow. A full `npm install` can take 2–5 minutes:
```bash
become my-tool npm install
```
### 5.3 Optimize Install Speed
- **Use `.npmrc` to avoid audit/lockfile overhead:**
```
audit=false
fund=false
lockfile=false
```
- **Pin exact versions** in `package.json` to avoid resolution time on every deploy
- **Install locally, copy binary:** `npm pack` on your machine, upload the `.tgz`, extract on Toolforge
- **Cache `node_modules`** — don't delete it on every deploy. Only re-run `npm install` when dependencies change.
### 5.4 When to Use Dependencies vs. Built-In
| Need | Recommendation |
|------|---------------|
| Simple REST API | Built-in `http` module — zero dependencies |
| Static file server | Built-in `http` + `fs/promises` |
| JSON body parsing | Built-in `JSON.parse()` + streaming body collection |
| URL parsing | Built-in `URL` class |
| Routing (simple) | `if/else` on `url.pathname` |
| Routing (complex, 10+ routes) | `express` or `fastify` |
| SQLite database | `better-sqlite3` |
| WebSocket | `ws` package |
| Templating | Built-in template literals or `ejs` |
**Rule of thumb:** Start with zero dependencies. Add packages only when the built-in solution becomes unwieldy.
---
## SOP 6: Environment Variables and Secrets
### Setting Variables
```bash
# Set variables — these persist across restarts
become my-tool toolforge env set API_KEY "sk-abc123"
become my-tool toolforge env set NODE_ENV "production"
become my-tool toolforge env set LOG_LEVEL "info"
# List all variables
become my-tool toolforge env list
# Remove a variable
become my-tool toolforge env unset API_KEY
```
### Reading Variables in Code
```javascript
const apiKey = process.env.API_KEY;
if (!apiKey) {
console.error('API_KEY not set — some features will be unavailable');
}
```
### Caveats
- **Variables are set per-tool**, not per-webservice-instance. All pods for the same tool share the same environment.
- **Restart the webservice after changing env vars** — they're loaded at process start, not dynamically:
```bash
become my-tool webservice --backend=kubernetes node22 restart
```
- **Never hardcode secrets** in source files committed to version control. Always use `toolforge env set`.
---
## SOP 7: Logging and Debugging
### Kubernetes Logs
```bash
# Tail logs from the current pod
become my-tool kubectl logs -f deployment/my-tool
# Last N lines
become my-tool kubectl logs --tail=100 deployment/my-tool
# Follow a specific pod (useful with multiple replicas)
become my-tool kubectl logs -f pod/my-tool-7d8f9e2a4-x3y7z
# Search for errors
become my-tool kubectl logs deployment/my-tool | grep -i error
```
### Structured Logging (Zero-Dependency)
```javascript
function log(level, message, data = {}) {
const entry = {
timestamp: new Date().toISOString(),
level,
message,
...data,
};
// Write to stdout — Kubernetes collects this
process.stdout.write(JSON.stringify(entry) + '\n');
}
// Usage
log('info', 'Server starting', { port: PORT });
log('error', 'Failed to resolve file', { file: filename, error: e.message });
```
### Common Issues
| Symptom | Likely Cause | Fix |
|---------|-------------|-----|
| `Connection refused` | Server not listening on `PORT` | Check `process.env.PORT` in code |
| `CrashLoopBackOff` in status | Server crashes on startup | Run `kubectl logs` to see the error |
| `ImagePullBackOff` | Runtime image not found | Check runtime name (`node22` not `nodejs22`) |
| Slow first request | Cold start + NFS metadata | Use a keepalive or health check endpoint |
| `ERR_FAILED 200 (OK)` | Streamed response to Pannellum | Use `readFile()` instead of streaming |
| 404 for existing files | Path resolution wrong | Log `filePath` and check directory traversal |
### Health Check Endpoint
Toolforge's Kubernetes expects the server to be responsive. Add a lightweight health check:
```javascript
server.on('listening', () => {
// Signal to Kubernetes that the server is ready
console.log('READY');
});
```
---
## SOP 8: Cron Jobs
For batch processing, periodic syncs, or scheduled maintenance:
```bash
# Run a script every hour
become my-tool job schedule node22 \
--command 'node /data/project/my-tool/cron/cleanup.mjs' \
--no-filelog \
"0 * * * *"
# Run once
become my-tool job submit node22 \
--command 'node /data/project/my-tool/cron/backfill.mjs'
```
### Cron Job Script Template
```javascript
// cron/cleanup.mjs — Scheduled script (no HTTP server needed)
import { readdir, unlink, stat } from 'node:fs/promises';
import { join } from 'node:path';
const CACHE_DIR = '/data/project/my-tool/images';
const MAX_AGE_DAYS = 7;
async function cleanup() {
const files = await readdir(CACHE_DIR);
const now = Date.now();
let removed = 0;
for (const file of files) {
const filePath = join(CACHE_DIR, file);
const stats = await stat(filePath);
const ageDays = (now - stats.atimeMs) / (1000 * 60 * 60 * 24);
if (ageDays > MAX_AGE_DAYS) {
await unlink(filePath);
removed++;
}
}
console.log(`Cleanup: removed ${removed} files (threshold: ${MAX_AGE_DAYS} days)`);
}
cleanup().catch(err => {
console.error('Cleanup failed:', err);
process.exit(1);
});
```
---
## SOP 9: Restarting and Updating
### Restart (zero-downtime)
```bash
become my-tool webservice --backend=kubernetes node22 restart
```
The Kubernetes deployment performs a rolling restart — no downtime as long as your server starts quickly.
### Update Code
```bash
# Upload new version
scp server.mjs <user>@login.toolforge.org:/data/project/my-tool/
# Restart to pick up changes
ssh <user>@login.toolforge.org "become my-tool webservice --backend=kubernetes node22 restart"
```
### Stop
```bash
become my-tool webservice --backend=kubernetes node22 stop
```
---
## Reference: Quick Command Cheat Sheet
```bash
# === Lifecycle ===
become my-tool webservice --backend=kubernetes node22 start
become my-tool webservice --backend=kubernetes node22 stop
become my-tool webservice --backend=kubernetes node22 restart
become my-tool webservice --backend=kubernetes node22 status
# === Logs ===
become my-tool kubectl logs -f deployment/my-tool
become my-tool kubectl logs --tail=100 deployment/my-tool
# === File Transfer (from local machine) ===
scp server.mjs <user>@login.toolforge.org:/data/project/my-tool/
scp -r public/ <user>@login.toolforge.org:/data/project/my-tool/
# === Environment ===
become my-tool toolforge env set KEY value
become my-tool toolforge env list
become my-tool toolforge env unset KEY
# === Scheduled Jobs ===
become my-tool job schedule node22 --command 'node script.mjs' "0 * * * *"
become my-tool job submit node22 --command 'node script.mjs'
# === Direct Shell ===
become my-tool
```
---
## Reference: Common Pitfalls
| Pitfall | Why It Happens | Prevention |
|---------|---------------|------------|
| **Missing PORT** | Hardcoded port in code | Always use `process.env.PORT || 8765` |
| **Streaming to Pannellum** | `createReadStream().pipe(res)` causes `ERR_FAILED` | Always use `readFile()` + `res.end()` for images |
| **No Cache-Control** | Missing headers = re-download on every page load | Always set cache headers for static assets |
| **Cached 404 pages** | 404 served with `Cache-Control: public` | Use `no-store` for error responses |
| **Slow npm install** | NFS metadata operations are slow | Minimize dependencies; use `.npmrc` to skip audit |
| **Missing MIME type** | `Content-Type` not set, browser refuses to load | Explicit MIME map for all file extensions |
| **Traversal vulnerability** | `join(dir, pathname)` can escape the root | Always `path.resolve()` and check `startsWith(dir + path.sep)` |
| **Crash on startup** | Unhandled promise rejection in `createServer` | Wrap async handlers in try/catch |
| **Stale image cache** | Commons file overwritten, old URL still cached | Append `?v={timestamp}` to cached image URLs |
| **CORS errors** | `upload.wikimedia.org` has no CORS headers | Proxy images through your own server |
---
## See Also
- **[wikimedia-toolforge](../wikimedia-toolforge/SKILL.md)** — Account setup, tool creation, SSH, Kubernetes pods, cron jobs, database access
- **SOP 9: Frontend Assets (Privacy-Preserving CDN)** — Must-load JS/CSS from `tools-static.wmflabs.org/cdnjs/` instead of external CDNs
- **[commons-file-resolution](../commons-file-resolution/SKILL.md)** — Resolving `File:` references to HTTP URLs, caching images for same-origin serving
- **[wikimedia-api-access](../wikimedia-api-access/SKILL.md)** — User-Agent, rate limiting, error handling
- **[wikimedia-commons-thumbnails](../wikimedia-commons-thumbnails/SKILL.md)** — Thumbnail URL generation and format conversion
- **[wikimedia-security-and-privacy](../wikimedia-security-and-privacy/SKILL.md)** — CDN privacy risks, data minimization, and third-party script policies