git:20260914.a10952c to git:20261003.928fb9a

---

name: wikimedia-toolforge

description: Manage Toolforge accounts, web services, Kubernetes pods, cron jobs, and file deployment for Wikimedia tools

license: MIT

compatibility: opencode

depends_on: [wikimedia-api-access]

skill_discovery_hints:

- keywords: ["Toolforge", "tool hosting", "Kubernetes", "web service", "cron job", "deploy"]

- keywords: ["toolsadmin", "become", "webservice", "toolforge jobs"]

- keywords: ["CDN", "cdnjs", "tools-static", "privacy-preserving CDN", "content delivery", "third-party script", "external CDN tracking"]

---

Toolforge (formerly Wikimedia Tool Labs) is a cloud hosting platform for community-developed tools that interact with Wikimedia wikis and data. This skill covers account setup, service management, deployment, and debugging.

## Prerequisites

- A [Toolforge account](https://toolforge.org/) approved via Wikimedia Foundation onboarding

- SSH public key added to your Toolforge account via the [admin console](https://admin.toolforge.org/)

- `ssh-agent` running locally with your private key loaded (`ssh-add ~/.ssh/id_ed25519`)

- The following environment variable set (optional but recommended):

- `TOOLFORGE_USER` — Your Toolforge shell/LDAP username

## SOP 1: Account & Project Setup

### 1.1 Verify Account

```bash

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org hostname

```

If successful, you will see a hostname like `tools-sgebastion-XX`. If the connection hangs or fails, check that your SSH key is added to the admin console.

### 1.2 Create a New Tool

Each tool is a sub-account with its own directory, database access, and web service.

Tool creation is **web-UI only** — the `toolforge tools ...` command family was

removed from the Toolforge CLI in 0.3.x and has **no CLI equivalent**. Create tools at

<https://toolsadmin.wikimedia.org/tools/create>.

> ⏳ **Wait ~1 minute after creation** before `become` works (see Common Mistakes).

**Naming rules:**

- Lowercase letters, digits, and hyphens only

- Must be unique across all Toolforge

- Should be descriptive (e.g., `pageview-analyzer`, `category-watchdog`)

After creation, the tool's home directory is at `/data/project/my-tool-name/`.

> ⚠️ **Toolforge Rule #2 — Open Source License Required:** All code in the Tools

> project must be published under an [OSI-approved](https://opensource.org/licenses)

> open source license. Add a `LICENSE` file to your repository before deploying.

> The absence of a license means default copyright laws apply, which is counter to

> the principles of the Wikimedia movement. See the

> [full rules](https://wikitech.wikimedia.org/wiki/Help:Toolforge/Rules).

### 1.3 Configure Tool Permissions

Maintainers are managed via the web UI at <https://toolsadmin.wikimedia.org/tools>

(open the tool, then "Maintainers"). The tool's creator is a maintainer automatically.

There is **no CLI equivalent** — `toolforge tools maintainers ...` does not exist.

## SOP 2: File Deployment

### 2.1 Deploy Files via SSH

```bash

# Deploy a single file

scp my-script.py ${TOOLFORGE_USER:-your-username}@login.toolforge.org:/data/project/my-tool-name/

# Deploy an entire directory

rsync -avz --exclude '.*' ./my-tool/ ${TOOLFORGE_USER:-your-username}@login.toolforge.org:/data/project/my-tool-name/

```

### 2.1 Copy Files via rsync/scp

> ⚠️ **One-shot commands over SSH — don't use `become`.** `become` is for

> interactive shells; over a chained SSH command (`ssh host "become X; cmd"`)

> it replaces the shell and the rest of the chain runs unbecome'd. For

> non-interactive one-shot commands use `sudo` directly:

>

> ```bash

> ssh ${TOOLFORGE_USER:-your-username}@dev.toolforge.org "sudo -niu tools.<tool-name> <command>"

> ```

>

> Also note: SSH as your **personal** account only — `tools.<tool>@dev.toolforge.org`

> is not an SSH login (fails with `Permission denied (publickey)`).

**Always use `rsync` for repeated deployments** — it only transfers changed files and preserves permissions.

### 2.2 Deploy via Git (Recommended for Maintained Tools)

```bash

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org

cd /data/project/my-tool-name/

git clone https://github.com/your-org/my-tool.git .

```

Then update with `git pull` on subsequent deployments. This provides version history and rollback.

### 2.3 Set Executable Permissions

```bash

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org chmod +x /data/project/my-tool-name/my-script.py

```

### 2.4 Fix File Ownership (the `take` command)

Files copied via `scp` or `rsync` arrive owned by your shell user, not the tool user.

Many tool commands require tool-user ownership:

```bash

# File landed as wrong user after scp/rsync

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org

become my-tool-name

take /data/project/my-tool-name/my-script.py

```

The `take` command changes ownership to the current tool user. It requires

that the calling user is the current owner (prevents abuse).

## SOP 3: Web Services

Toolforge supports several web service backends. Choose based on your needs:

| Backend | Use Case | Start Command |

|---------|----------|---------------|

| `webservice --backend=kubernetes python3.11` | Python web apps (Flask, Django, FastAPI) | `webservice --backend=kubernetes python3.11 start` |

| `webservice --backend=kubernetes node` | Node.js web apps (Express) | `webservice --backend=kubernetes node start` |

| `webservice --backend=kubernetes php8.2` | PHP web apps | `webservice --backend=kubernetes php8.2 start` |

| `webservice --backend=kubernetes static` | Static file serving (HTML/JS) | `webservice --backend=kubernetes static start` |

| `webservice --backend=buildservice` | Build Service container (any language) | `webservice --backend=buildservice start` |

### 3.1 Start a Web Service

```bash

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org

become my-tool-name

cd /data/project/my-tool-name

webservice --backend=kubernetes python3.11 start

```

The `become` command switches to the tool's service account, which has the correct permissions and environment variables.

### 3.2 Check Service Status

```bash

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org

become my-tool-name

webservice --backend=kubernetes python3.11 status

```

### 3.3 Stop a Web Service

```bash

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org

become my-tool-name

webservice --backend=kubernetes python3.11 stop

```

### 3.4 Restart a Web Service

```bash

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org

become my-tool-name

webservice --backend=kubernetes python3.11 restart

```

### 3.5 Web Service Entry Point

The web service looks for a `server.py` (Python), `app.js` (Node), or `index.php` (PHP) in the tool's home directory by default. To use a custom entry point, set the `WEB_CONCURRENCY` or create a `launch.sh` script:

```bash

#!/bin/bash

# launch.sh — custom web service entry point

cd /data/project/my-tool-name

gunicorn -w 4 -b 0.0.0.0:8000 my_app:app

```

Mark it executable: `chmod +x launch.sh`, then start with:

```bash

webservice --backend=kubernetes python3.11 start --launch launch.sh

```

## SOP 4: Kubernetes Jobs (One-Off Tasks)

For one-off or batch processing tasks that don't need a persistent web service:

```bash

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org

become my-tool-name

toolforge jobs run my-job-name --command "python3 /data/project/my-tool-name/my_script.py" --image python3.11 --wait

```

### Common Job Options

| Option | Description |

|--------|-------------|

| `--command "..."` | Command to run inside the container |

| `--image python3.11` | Container image (choose based on language/runtime) |

| `--wait` | Wait for the job to finish before returning logs |

| `--mem 2Gi` | Memory limit (default: 1Gi, max: 4Gi) |

| `--cpu 1` | CPU cores (default: 1, max: 2) |

| `--filelog` | Stream logs to a file on NFS (for later inspection) |

| `--timestamps` | Add timestamps to log output |

### Check Job Status

```bash

become my-tool-name

toolforge jobs list

toolforge jobs logs my-job-name

```

### Delete a Job

```bash

become my-tool-name

toolforge jobs delete my-job-name

```

### NFS Writes from Inside Pods (Non-Root Caveats)

Kubernetes job containers run as the tool user, not root. The NFS home at

`/data/project/<tool>/` is writable, but system directories are not:

| Can write to | Cannot write to |

|--------------|----------------|

| `/data/project/<tool>/` (NFS home) | `/usr/local/` (root owned) |

| `/tmp/` (world-writable) | `/var/lib/dpkg/` (root owned) |

| `$HOME` (on NFS) | `/etc/` (root owned) |

| `$TOOL_DATA_DIR` | `/data/` (root-owned) |

**Consequences:**

- Cannot `apt-get install` anything

- Cannot `npm install -g` without setting `NPM_CONFIG_PREFIX` to a writable dir

- Cannot write to `/data/` — use `/data/project/<tool>/` or `$TOOL_DATA_DIR` instead

- `kubectl cp` may fail with permission errors on the kubeconfig file

(use `kubectl exec -i ... -- sh -c 'cat > /path' < localfile` instead)

## SOP 5: Cron Jobs

For scheduled, recurring tasks:

```bash

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org

become my-tool-name

crontab -e

```

Add a line in standard cron format:

```

# Run daily at 2:00 AM UTC

0 2 * * * /usr/bin/python3 /data/project/my-tool-name/daily_report.py >> /data/project/my-tool-name/logs/cron.log 2>&1

```

**Cron job guardrails:**

- Always redirect output to a log file (otherwise cron emails it to you and can fill your mailbox)

- Use absolute paths for all commands and file references

- Test the script manually before scheduling it via cron

- Set a reasonable schedule — do not run on sub-minute intervals

- For long-running jobs (over 10 minutes), use `toolforge jobs` (SOP 4) triggered by cron instead of running directly

### List Cron Jobs

```bash

become my-tool-name

crontab -l

```

## SOP 6: Environment Variables & Secrets

### 6.1 Set Environment Variables

Set tool-scoped environment variables with `envvars create` (run as the tool):

```bash

become my-tool-name

toolforge envvars create MY_VARIABLE my_value

```

### 6.2 View Environment Variables

```bash

become my-tool-name

toolforge envvars list

```

Values are hidden by default; add `--show-values` to display them.

### 6.3 Remove an Environment Variable

```bash

become my-tool-name

toolforge envvars delete MY_VARIABLE

```

### 6.4 Secrets (Database Passwords, API Keys)

Store sensitive values as environment variables via `toolforge envvars create`. These are stored securely and not shown in `envvars list` output by default. Do not hardcode secrets in source files.

### 6.5 MediaWiki API Authentication (Bot Passwords)

If your Toolforge tool authenticates against the MediaWiki API to make edits

(e.g., via bot passwords or Pywikibot), be aware that the API's `lgname`

parameter requires **underscores** where usernames have spaces:

```python

# ✗ Fails silently with "Unknown error"

lgname = "AL Wiki MIT@mybot"

# ✓ Works

lgname = "AL_Wiki_MIT@mybot"

```

See the **[pywikibot](../pywikibot/SKILL.md)** skill ("Login fails" troubleshooting

section) and the **[wikimedia-api-access](../wikimedia-api-access/SKILL.md)** skill

("Login Username Quirk" section) for full details.

## SOP 7: Logs & Debugging

### 7.1 Web Service Logs

```bash

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org

become my-tool-name

webservice --backend=kubernetes python3.11 logs

```

Add `--tail` to see only the last N lines:

```bash

webservice --backend=kubernetes python3.11 logs --tail=50

```

### 7.2 Job Logs

```bash

become my-tool-name

toolforge jobs logs my-job-name

```

### 7.3 Check Disk Usage

```bash

become my-tool-name

du -sh /data/project/my-tool-name/

```

### 7.4 Interactive Debugging via Shell

```bash

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org

become my-tool-name

# Now run commands interactively from the tool's home directory

python3 -c "import requests; print(requests.get('https://en.wikipedia.org/api/rest_v1/page/summary/Python_(programming_language)').json())"

```

## SOP 8: Build Service (Container Images)

The Toolforge Build Service allows you to build custom container images from a public Git

repository using Cloud Native Buildpacks. This is the modern, recommended way to deploy

tools — it frees you from per-language base images and gives you control over the runtime.

### 8.1 How It Works

Instead of deploying files to NFS and using a language-specific webservice backend, you:

1. Host your code in a **public Git repository** (GitLab, GitHub, Gerrit)

2. Add a **`Procfile`** at the root of your repo defining how to start your app

3. Run `toolforge build start` to build a container image from your repository

4. Run the built image as a **web service** or a **job**

The Build Service supports: Python, Node.js, PHP, Ruby, Go, Java/JVM, .NET, and Rust.

It can also install OS-level Apt packages and compile frontend assets with Node.js at build time.

### 8.2 Prerequisites

- A **public Git repository** containing your tool's code

- A **Procfile** at the repository root

- `become` access to your tool account

### 8.3 Procfile Format

The Procfile is a plain text file named `Procfile` (no extension) at the root of your repo:

```

web: gunicorn --bind=0.0.0.0 --workers=4 --forwarded-allow-ips=* app:app

migrate: python -m app.django migrate

```

- The `web:` entry defines what runs when you start a **web service**

- Other entries (e.g., `migrate:`) define what runs as **jobs**

- Every process type in the Procfile becomes an executable command in the container

**Important:** Do not name a process type after a real command (e.g., do not use `celery:`

as a process type — use `run-celery:` instead).

### 8.4 Building an Image

```bash

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org

become my-tool-name

# Build from the default branch (HEAD)

toolforge build start https://gitlab.wikimedia.org/toolforge-repos/my-tool

# Build from a specific branch, tag, or commit

toolforge build start --ref v1.2.0 https://gitlab.wikimedia.org/toolforge-repos/my-tool

# Use the latest buildpacks (newer language versions)

toolforge build start -L https://gitlab.wikimedia.org/toolforge-repos/my-tool

# Pass build-time environment variables

toolforge build start --envvar NODE_ENV=production https://gitlab.wikimedia.org/toolforge-repos/my-tool

```

### 8.5 Starting a Web Service

```bash

become my-tool-name

# Start the web service from the built image

toolforge webservice buildservice start --mount=none

# Give memory-hungry apps headroom (default limit is 512 Mi — see gotchas below)

toolforge webservice buildservice start --mount=none -m 1Gi

# Alternatively, create a service.template so 'webservice start' works directly:

echo -e 'type: buildservice\nmount: none' > service.template

# Check status

toolforge webservice buildservice status

# View logs

toolforge webservice buildservice logs -f

# Restart after a new build

toolforge webservice buildservice restart

# Stop

toolforge webservice buildservice stop

# Get an interactive shell inside the running container

toolforge webservice buildservice shell

```

**Build Service gotchas (field-tested 2026-08-14, Reframe Flask+OpenCV tool):**

- **`runtime.txt` is dead** — heroku/python buildpacks ≥ 6.x fail the build

with `Error: The runtime.txt file isn't supported`; use `.python-version`

containing just the major.minor (e.g. `3.11`).

- **Default pod memory limit is 512 Mi** — each gunicorn sync worker is a

separate process (~100–150 MB RSS with OpenCV/numpy/ML libraries), so

`--workers=4` under 512 Mi = OOMKilled container. Raise with `-m 1Gi` at

start; budget workers × per-process RSS against the limit.

- **OOM symptom:** intermittent `upstream connect error or disconnect/reset

before headers. reset reason: connection termination` from the ingress.

Diagnose with `kubectl get pod <pod> -o jsonpath='{.status.

containerStatuses[0].lastState.terminated.reason}'` (→ `OOMKilled`),

`kubectl top pods` (live RSS), and `kubectl get deployment -o jsonpath='{.spec.

template.spec.containers[0].resources}'` (limit).

- **Build source:** any public Git repo works — GitLab, **GitHub**

(verified), or Gerrit: `toolforge build start https://github.com/<user>/<repo>`.

- gunicorn 26 logs a harmless `Control server error: [Errno 13] Permission

denied` at container startup — non-fatal, ignore.

**Note on NFS mounts:** By default, `--mount=none` is recommended. If your tool needs

to read/write files in `/data/project/`, use `--mount=all` and reference the path via

the `$TOOL_DATA_DIR` environment variable instead of relying on `$HOME`.

### 8.6 Running a Job

```bash

become my-tool-name

# Run the 'migrate' process from your Procfile

toolforge jobs run --wait --image my-tool/my-tool:latest --command "migrate" some-job

# Run with arguments

toolforge jobs run --wait --image my-tool/my-tool:latest --command "migrate --production" migrate-job

# Pass composite commands via shell wrapper

toolforge jobs run --wait --image my-tool/my-tool:latest --command "sh -c 'env; nodejs --version'" debug-job

```

### 8.7 Checking Build Logs

```bash

become my-tool-name

# View the most recent build log

toolforge build logs

# Check build quota

toolforge build quota

```

### 8.8 Updating Code

To deploy a new version:

```bash

become my-tool-name

# 1. Push new code to your Git repo

# 2. Trigger a new build

toolforge build start https://gitlab.wikimedia.org/toolforge-repos/my-tool

# 3. Restart the web service to use the new image

toolforge webservice buildservice restart

```

### 8.9 Installing Apt Packages

Create a `project.toml` at the root of your repository:

```toml

[_]

schema-version = "0.2"

[com.heroku.buildpacks.deb-packages]

install = [

"imagemagick",

"php",

]

```

Packages from Ubuntu 24.04 (Noble) can be looked up at

https://packages.ubuntu.com/noble/.

### 8.10 Known Limitations

- Limited to a single primary language runtime per image (Node.js can be a secondary runtime for asset compilation)

- No LDAP connection inside containers — commands like `id <user>` will not work

- `$HOME` does not point to `/data/project/<tool>/` — use `$TOOL_DATA_DIR` instead

- NFS is not mounted by default — use `--mount=all` explicitly when needed

- Build images have a storage quota — check with `toolforge build quota` and request increases via Phabricator if needed

## Guardrails & Common Pitfalls

1. **Use `become` for interactive sessions, `sudo -niu` for SSH one-liners.**

Running `become my-tool-name` interactively (SSH in, then `become`, then commands)

works correctly. But `become` uses `exec` internally, which **replaces the shell** —

so command chaining over SSH fails:

```bash

# ❌ BROKEN — second command runs as YOUR user, not the tool

ssh user@login.toolforge.org "become my-tool-name; webservice restart"

# ✅ CORRECT — use sudo -niu for chained SSH commands

ssh user@login.toolforge.org "sudo -niu tools.my-tool-name webservice restart"

```

2. **SSH key expiry** — Toolforge SSH keys expire after a period. If you get permission denied, regenerate your key in the admin console and re-add it to `ssh-agent`.

3. **NFS latency** — `/data/project/` is on NFS. File operations can be slow. Avoid frequent small writes. Use local `/tmp/` for temporary files and move results to NFS only when needed.

4. **Resource limits** — Kubernetes pods have 1 CPU and 1Gi RAM by default. Use toolforge jobs with `--mem` and `--cpu` flags for larger tasks. Do not run resource-intensive tasks on bastion or login nodes.

5. **Do not run long processes on login** — The login shell is for administration only. Long-running processes should be jobs or web services. Processes running for more than 30 minutes on login may be killed without warning.

6. **Database connections** — For replica database access, see the `wikimedia-database` skill. For tool-owned databases (MariaDB), use `become my-tool-name` and run `sql my-tool-name` to access the tool's database. If you need the actual MySQL username and password (e.g., for an external client or connection string), find them in the tool's home directory after `become`:

```bash

become my-tool-name

cat replica.my.cnf

```

This prints `[client]` with `user` and `password` fields. The `sql` command is still the recommended way to connect interactively, but `replica.my.cnf` is useful when configuring ORM connection strings or database drivers in application code.

7. **Static file caching** — Static web services (`--backend=kubernetes static`) serve from `/data/project/my-tool-name/`. Files are cached; wait a few minutes after deployment or use a versioned URL pattern (`style.v2.css`).

8. **Test locally first** — Deploying broken code to Toolforge wastes time. Test scripts locally with representative data before deploying.

9. **Clean up old jobs** — Kubernetes job history accumulates. Delete completed jobs that are no longer needed using `toolforge jobs delete`.

10. **Build Service: Git repo required** — The Build Service requires a **public** Git repository. Private repos are not supported. The `Procfile` must be at the repository root and named exactly `Procfile` (no extension). After a build, you must restart the web service to pick up the new image — `toolforge build start` alone does not restart running services.

11. **Build Service: NFS and $HOME** — Build Service containers do not have NFS mounted by default. Use `--mount=all` to mount it. Inside the container, `$HOME` does not point to `/data/project/<tool>/` — use the `$TOOL_DATA_DIR` environment variable instead for tool home directory paths.

12. **Wait ~1 minute after tool creation** before `become` works. Use `sudo -u` as an immediate workaround.

13. **Refs vary between browser sessions** — When using Playwright or browser automation for toolsadmin, always snapshot to read the current accessibility tree instead of hardcoding element refs.

### Common Toolforge Mistakes

| Mistake | Symptom | Fix |

|---------|---------|-----|

| `become <tool>; cmd1; cmd2` | Subsequent commands run as original user | Use `sudo -niu tools.<tool> cmd1; cmd2` instead |

| `become <tool>; webservice restart` | TLS cert errors or commands run as wrong user | Use `sudo -niu tools.<tool> webservice restart` |

| `--kubeconfig=PATH` | `stat .kubeconfig: no such file` | Use `--kubeconfig PATH` (space, not `=`) |

| `toolforge env set` | `No such command 'env'` | Use `toolforge envvars create` |

| Nested `"` inside `"` over SSH | `unexpected EOF` | Use heredoc or alternate quote layers |

| `scp` to `/data/project/` | `Permission denied` (lands as wrong user) | Pipe through `sudo`; use `take` |

| `kubectl exec -it` in non-tty | `the input device is not a TTY` | Use `-i` without `-t`, or add `-t` to the outer SSH command |

| `kubectl cp` to a running pod | Permission errors reading kubeconfig | Use `kubectl exec -i ... -- sh -c 'cat > /path' < localfile` instead |

| Writing to `/data/` from inside a pod | Permission denied | Use `/data/project/<tool>/` or `$TOOL_DATA_DIR` instead |

| `printf '\\e'` for escape sequences | `\e` becomes literal escape char (0x1B) | Use `printf '%s' '\\e'` with `%s` format to output literally |

| `become <tool> sh -c "\$VAR"` | Variable empty on bastion | Escaped `$` passes through to the tool shell; unescaped expands on bastion |

## Example Workflows

### Full Web App Deployment

```bash

# 1. Create tool (web UI only: https://toolsadmin.wikimedia.org/tools/create)

# No CLI equivalent — `toolforge tools` was removed in CLI 0.3.x.

# 2. Deploy code

rsync -avz ./my-web-app/ user@login.toolforge.org:/data/project/my-web-tool/

# 3. Set environment variables

ssh user@login.toolforge.org "become my-web-tool; toolforge envvars create API_KEY your-api-key-here"

# 4. Start web service

ssh user@login.toolforge.org "become my-web-tool; webservice --backend=kubernetes python3.11 start"

# 5. Verify it's running

ssh user@login.toolforge.org "become my-web-tool; webservice --backend=kubernetes python3.11 status"

```

### Daily Data Collection (Cron + Job)

```bash

# 1. Deploy the script

rsync -avz collect_data.py user@login.toolforge.org:/data/project/my-tool/

# 2. Set cron to trigger a Kubernetes job

ssh user@login.toolforge.org "become my-tool; crontab -l | { cat; echo '0 3 * * * toolforge jobs run daily-collect --command \"python3 /data/project/my-tool/collect_data.py\" --image python3.11 --wait --filelog >> /data/project/my-tool/logs/cron_trigger.log 2>&1'; } | crontab -"

```

### Build Service Web App (Modern Workflow)

```bash

# 1. Create tool (web UI only: https://toolsadmin.wikimedia.org/tools/create)

# No CLI equivalent — `toolforge tools` was removed in CLI 0.3.x.

# 2. Push code to a public Git repository (e.g., GitLab)

# Ensure a Procfile exists at the root:

# web: gunicorn --bind=0.0.0.0 --workers=4 app:app

# 3. Build the container image from the Git repo

ssh user@login.toolforge.org "become my-build-tool; toolforge build start https://gitlab.wikimedia.org/toolforge-repos/my-build-tool"

# 4. Start as a build service web service

ssh user@login.toolforge.org "become my-build-tool; toolforge webservice buildservice start --mount=none"

# 5. Verify it's running

ssh user@login.toolforge.org "become my-build-tool; toolforge webservice buildservice status"

# 6. To update: push new code to Git, rebuild, restart

# ssh login.toolforge.org "become my-build-tool; toolforge build start <repo-url>"

# ssh login.toolforge.org "become my-build-tool; toolforge webservice buildservice restart"

```

---

## Tooling

This skill includes helper scripts, reference docs, and templates:

### 🔧 Deploy Tool (`scripts/deploy.sh`)

Deploy files to a Toolforge tool via rsync with dry-run preview.

```bash

./scripts/deploy.sh ./my-web-app my-tool-name

```

Features dry-run confirmation, permission setting, and post-deploy steps.

**Note:** For the Build Service (SOP 8), you do not use rsync deployment. Instead,

push code to a public Git repository and use `toolforge build start`. See the

Build Service workflow example above.

### 🔧 Status Check (`scripts/status.sh`)

Check web service status, Kubernetes jobs, disk usage, and active processes.

```bash

./scripts/status.sh my-tool-name

```

### 🔧 Kubernetes Job Manager (`scripts/manage-k8s.sh`)

Manage Kubernetes jobs: run, list, logs, delete, and status.

```bash

./scripts/manage-k8s.sh my-tool-name run my-job "python3 /data/project/my-tool/script.py"

./scripts/manage-k8s.sh my-tool-name list

./scripts/manage-k8s.sh my-tool-name logs my-job

```

### 🔧 Cron Job Manager (`scripts/manage-cron.sh`)

Manage cron jobs: list, add, remove by pattern, or clear all.

```bash

./scripts/manage-cron.sh my-tool-name list

./scripts/manage-cron.sh my-tool-name add '0 2 * * *' 'python3 /data/project/my-tool/daily.py >> /data/project/my-tool/logs/cron.log 2>&1'

./scripts/manage-cron.sh my-tool-name remove daily.py

```

### 📚 CLI Reference (`references/toolforge-cli.md`)

Quick reference of all Toolforge CLI commands organized by category:

- Account & tools, web services (including buildservice), Kubernetes jobs, environment variables, cron, file operations, database

### 🧩 Deploy Config (`assets/deploy-config.sh`)

Environment variable template for deployment scripts:

```bash

cp assets/deploy-config.sh my-config.sh

# Edit my-config.sh with your Toolforge username and tool name

source my-config.sh

./scripts/deploy.sh ./my-app my-tool-name

```

### 🐍 Flask App Template (`assets/app-template.py`)

Ready-to-deploy Flask app with:

- Home page with status

- Health check endpoint (`/api/status`)

- Wikipedia API proxy (`/api/summary/<title>`, `/api/search?q=...`)

- Proper User-Agent for Wikimedia API requests

- WSGI entry point for gunicorn

```bash

cp assets/app-template.py server.py

# Edit, test locally with `python3 server.py`, then deploy

```

**For Build Service deployment:** Add a `Procfile` at your repo root containing:

```

web: gunicorn --bind=0.0.0.0 --workers=4 --forwarded-allow-ips=* app:app

```

Then push to a public Git repo and run `toolforge build start`.

---

## SOP 9: Frontend Assets (Privacy-Preserving CDN)

Toolforge tools must load JavaScript, CSS, and fonts from Wikimedia's internal

cdnjs mirror — never from external CDNs that track users.

### Rationale

External CDNs (cdnjs.cloudflare.com, unpkg.com, Google Fonts) can log IPs and

track users. Wikimedia's **Toolforge Web Hosting Policy** requires the internal

mirror for privacy compliance.

### CDN URLs

Only one hostname works reliably:

```

https://tools-static.wmflabs.org/cdnjs/ajax/libs/<library>/<version>/<file>

```

**Examples:**

```html

<script src="https://tools-static.wmflabs.org/cdnjs/ajax/libs/d3/7.9.0/d3.min.js"></script>

<script src="https://tools-static.wmflabs.org/cdnjs/ajax/libs/jquery/3.6.0/jquery.min.js"></script>

<link rel="stylesheet" href="https://tools-static.wmflabs.org/cdnjs/ajax/libs/twitter-bootstrap/5.3.0/css/bootstrap.min.css">

```

### Finding Libraries

Search the cdnjs API to find versions, then construct the mirror URL:

```bash

# Search for a library

curl -s "https://api.cdnjs.com/libraries?search=d3&fields=version,latest"

# → {"name":"d3","version":"7.9.0",...}

# Construct mirror URL by replacing cdnjs.cloudflare.com/ajax/libs

# with tools-static.wmflabs.org/cdnjs/ajax/libs

```

### Guardrails

| Do | Don't |

|----|-------|

| Use `tools-static.wmflabs.org/cdnjs/` | Use `cdnjs.cloudflare.com`, `unpkg.com`, `jsdelivr.net` |

| Pin exact versions (`7.9.0`) | Use `latest` or version-agnostic URLs |

| Verify availability before deploying | Assume all cdnjs libraries are mirrored |

The full CDN mirror guide with troubleshooting is at **[references/cdn-mirror-guide.md](./references/cdn-mirror-guide.md)**.

### Tooling

| File | Purpose |

|------|---------|

| [`scripts/check-cdn.sh`](./scripts/check-cdn.sh) | Verify a library is on the mirror (by name or full URL) |

| [`scripts/list-available.sh`](./scripts/list-available.sh) | Search available libraries |

| [`assets/load-template.html`](./assets/load-template.html) | HTML page loading jQuery, Bootstrap, Font Awesome from CDN |

| [`assets/load-template.js`](./assets/load-template.js) | Dynamic JS loader for programmatic use |

---

## Scripting Guide: Multi-Layer SSH Commands

Running commands on Toolforge involves nested shells: local → SSH → bastion → `become`/`sudo` → command. Each layer adds quoting complexity.

### The Shell Stack

```

Local machine ──▶ SSH ──▶ Bastion shell ──▶ sudo -niu tools.<tool> ──▶ Command

```

Every `"`, `$`, `\\`, `` ` ``, and `'` must survive all layers.

### Pattern 1: Simple Commands

```bash

# No special characters — works directly

ssh user@login.toolforge.org "sudo -niu tools.mytool toolforge jobs list"

# Single-quoted arguments inside the remote string

ssh user@login.toolforge.org "sudo -niu tools.mytool toolforge jobs run myjob --command 'python3 script.py'"

```

### Pattern 2: The `take` Command Alternative for File Transfer

You can't `scp` directly to `/data/project/<tool>/` as your shell user

(it's owned by `tools.<toolname>`). Two approaches:

**A) Staging area + `take`:**

```bash

scp file.txt user@login.toolforge.org:/home/user/

ssh user@login.toolforge.org \

"sudo -niu tools.mytool sh -c 'cp /home/user/file.txt /data/project/mytool/ && take /data/project/mytool/file.txt'"

```

**B) Pipe through SSH:**

```bash

cat localfile | ssh user@login.toolforge.org \

"sudo -niu tools.mytool sh -c 'cat > /data/project/mytool/localfile'"

```

### Pattern 3: Passing stdin to a Remote Command

```bash

# Pipe a local file into a remote command as the tool user

cat my-script.sh | ssh user@login.toolforge.org \

"sudo -niu tools.mytool sh -c 'cat > /data/project/mytool/my-script.sh'"

# Heredoc piped through SSH

cat << 'EOF' | ssh user@login.toolforge.org \

"sudo -niu tools.mytool sh -c 'cat > /data/project/mytool/config.json'"

{

"key": "value"

}

EOF

```

**Key insight for heredocs:** If you quote the delimiter (`<< 'EOF'`), the content is treated as a literal string — no variable expansion happens locally. This is usually what you want when writing to remote files.

### Pattern 4: Writing Files Inside a Pod (kubectl exec)

More reliable than `kubectl cp` (avoids kubeconfig permission issues):

```bash

# Pipe local content into a file inside a pod

cat config.json | kubectl exec -i my-pod -n tool-mytool -- sh -c 'cat > /tmp/config.json'

# Heredoc works the same way

cat << 'EOF' | kubectl exec -i my-pod -n tool-mytool -- sh -c 'cat > /tmp/.env'

API_KEY=sk-...

EOF

```

### Pattern 5: Commands with Internal Quotes

```bash

# Outer single quotes let you use double quotes inside for the remote command

ssh user@login.toolforge.org \

'sudo -niu tools.mytool kubectl exec pod -n tool-mytool -- sh -c "echo hello"'

```

### The Bash Quoting Idiom Trap

The idiom `'single'\''quote'` embeds a single quote inside a single-quoted string:

```bash

# IDIOM: 'text1'\''text2' produces: text1'text2

# ^^^^^^ ^^ ^^^^^^

# quoted \' quoted

# (escaped single quote between two quoted parts)

```

**Common mistake — space after `\'`:**

```bash

# WRONG: 'text1'\'' text2'

# The space after \' STARTS a new argument

# Same for 'text1\' 'text2'

# RIGHT: 'text1'\''text2' — no space, all one argument

```

**Safer alternative:** build strings in multiple steps:

```bash

printf '%s' 'PS1=' >> /tmp/file # PS1=

printf '%s' '\\[\\e...' >> /tmp/file # the actual PS1 value

printf '%s' "'" >> /tmp/file # closing single quote

echo "" >> /tmp/file # newline

```

### Writing Scripts That Run on the Bastion (No Local SSH Layer)

When you write a script that runs on the bastion itself (not from your local machine),

there's one fewer shell layer, which is much simpler:

```bash

# /data/project/mytool/myscript.sh (runs as tools.mytool)

# Simple: direct kubectl with --kubeconfig (use SPACE, not =)

kubectl --kubeconfig /data/project/mytool/.kube/config get pods -n tool-mytool

# Multi-line script inside pod (escape $ for variables you want expanded IN the pod)

kubectl exec my-pod -n tool-mytool -- sh -c "

export PATH=/tmp/npm/bin:\$PATH

node --version

"

```

**Critical rule for bastion scripts:**

- Variables you want expanded ON THE BASTION use `${VAR}` (no escaping)

- Variables you want expanded INSIDE THE POD use `\${VAR}` or `\$VAR` (escaped `$`)

---

## Cross-References

| Related Skill | Why |

|--------------|-----|

| **[wikimedia-database](../wikimedia-database/SKILL.md)** | SQL replicas — SSH tunnel setup shared with Toolforge |

| **[toolforge-nodejs](../toolforge-nodejs/SKILL.md)** | Node.js web services on Toolforge Kubernetes |

| **[toolforge-python](../toolforge-python/SKILL.md)** | Python web services on Toolforge Kubernetes (Flask + gunicorn) |

| **[wikimedia-codex](../wikimedia-codex/SKILL.md)** | Codex design system — components, design tokens, and icons for tool frontends |

| **[wikimedia-eventstreams](../wikimedia-eventstreams/SKILL.md)** | Real-time streams for monitoring tool events |

| **[wikimedia-i18n-l10n-for-tools](../wikimedia-i18n-l10n-for-tools/SKILL.md)** | Multilingual design for Toolforge tools |

| **[wikimedia-ml-services](../wikimedia-ml-services/SKILL.md)** | ML inference (Lift Wing) — often deployed as Toolforge services |

| **[wikimedia-phabricator](../wikimedia-phabricator/SKILL.md)** | Report Toolforge bugs and track feature requests |

| **[wikimedia-security-and-privacy](../wikimedia-security-and-privacy/SKILL.md)** | Secrets management and data minimization for Toolforge tools |

| **[pywikibot](../pywikibot/SKILL.md)** | Bot framework — often deployed on Toolforge |