v1.0.0 to v2.0.0
148 added, 18 removed. Audit A to A.
---
name: centralized-secrets-management-vault-integration
- description: Secure, centralized API key management using HashiCorp Vault. Enforces
- AppRole authentication, environment isolation, and secret caching for trading bots.
+ description: Use when a trading process must obtain exchange API keys or database
+ credentials from HashiCorp Vault instead of .env files — AppRole login, KV v2
+ reads, explicit token TTL/max-TTL handling, a TTL-bounded in-memory cache so a
+ rotated credential is actually picked up, and a redacting secret container that
+ keeps keys out of logs.
domain: Infrastructure
subdomain: Security
tags:
- vault
- hashicorp
- secrets
- api-keys
- approle
- security
+ - kv-v2
+ - token-lifetime
brokers_frameworks:
- HashiCorp Vault
- version: "1.0.0"
+ - HashiCorp Vault KV v2
+ version: "2.0.0"
author: algo-trading-skills-contributors
license: Apache-2.0
---
## When to Use
- Use this skill to completely eliminate hardcoded API keys, environment variables (.env files), or unencrypted config files in your trading infrastructure. A centralized Vault handles the storage, encryption, access control, and auditing of all sensitive data (Exchange API keys, Database passwords, Cloud tokens).
+ Invoke this skill when a trading process needs **live exchange or database credentials at
+ runtime** and you want them to come from Vault rather than a `.env` file, a config blob,
+ or the process image. `VaultSecretsManager` covers the client half of the AppRole
+ workflow:
+ 1. **Login** — `POST auth/approle/login` with a RoleID and SecretID, recording the
+ returned `lease_duration`, `renewable`, and `accessor`.
+ 2. **Read** — `GET {mount}/data/{path}` (KV v2), returning a `SecretBundle` that carries
+ the values plus `metadata.version` so a caller can see when a secret was rotated.
+ 3. **Hold** — a TTL-bounded cache so the process does not read Vault on every order, and
+ an explicit token lifetime so the token is renewed (or the process re-authenticates)
+ *before* a read fails.
+
+ The design assumption throughout is a **long-lived process**: a bot that boots once and
+ runs for days. That is precisely the case where a naive client breaks — its token silently
+ passes max TTL, or its unbounded cache keeps feeding a credential the security team
+ revoked hours ago.
+
+ ## When NOT to Use
+
+ - **You need Vault configured, not read.** Policies, AppRole provisioning, SecretID
+ delivery, and audit-device setup are operator work; see `references/workflows.md`. This
+ module authenticates and reads, nothing else.
+ - **You expect the client to be the access-control boundary.** It is not. The
+ `environment` guard rejects a malformed or wrong-environment *path string* before it
+ leaves the process; it cannot constrain a token whose Vault policy is too broad. If the
+ policy grants `secret/data/*`, this class will happily read whatever you ask it for
+ inside its own environment prefix.
+ - **You are rotating a credential, not fetching one.** The hot-swap, dual-credential
+ overlap, and revocation sequence belong to `secrets-rotation-without-bot-downtime`.
+ This skill's contribution to rotation is bounding staleness: `cache_ttl` and
+ `invalidate()`.
+ - **You are auditing what a key is permitted to do at the broker.** That is
+ `api-key-least-privilege-audit-tool`.
+ - **The secret must never exist in process memory.** Vault KV hands you plaintext. For
+ keys that must not leave a boundary, the operation must move to the key — see
+ `hardware-security-module-hsm-for-signing-keys`.
+ - **You need Vault's dynamic secrets or leases.** This client reads static KV v2 and does
+ not track or renew secret leases; dynamic database credentials need lease renewal and
+ revocation logic this module does not implement.
+
## Prerequisites
- - A running instance of HashiCorp Vault (or a compatible KMS like AWS Secrets Manager).
- - Trading bots must be configured with an `AppRole` Role ID and Secret ID.
- - The `hvac` Python library (or standard `requests` if wrapping the API directly).
+ - A reachable Vault server over **HTTPS**. `HttpVaultTransport` refuses `http://` unless
+ `allow_insecure_http=True`, because the token and every secret would otherwise cross
+ the network in clear text.
+ - A KV **v2** mount (`{mount}/data/{path}` reads). KV v1 has no `data`/`metadata`
+ envelope and this client will not parse it.
+ - An AppRole whose policy is scoped to exactly the paths this process needs, and whose
+ RoleID and SecretID arrive by **different channels** — HashiCorp's AppRole guidance
+ treats delivering both together as an anti-pattern, and recommends response-wrapping
+ the SecretID with `secret_id_num_uses=1`.
+ - A decision, made before deployment, about what happens when the SecretID is spent: with
+ `secret_id_num_uses=1`, re-login after max TTL fails permanently and an orchestrator
+ must deliver a fresh wrapped SecretID.
+ - Python 3.8+. No third-party package required; `hvac` can be substituted behind the
+ `VaultTransport` protocol.
## Workflow
- 1. **Vault Configuration**: The security team provisions an `AppRole` for a specific bot (e.g., `binance-market-maker-bot`). The policy restricts access strictly to the `secret/data/prod/binance/market-maker` path.
- 2. **Bot Initialization**: The trading bot boots and uses its `role_id` and `secret_id` to authenticate with Vault, receiving a temporary client token.
- 3. **Secret Retrieval**: The bot requests the required API keys using the token.
- 4. **Caching**: The bot caches the keys in protected memory (RAM only) so it does not hammer the Vault API on every single network request.
- 5. **Execution**: The bot injects the keys directly into the Exchange API client (e.g., CCXT or proprietary FIX engine).
+ 1. **Construct with the environment this process owns.**
+ `VaultSecretsManager("https://vault.internal:8200", "prod", mount="secret")`. The
+ environment is a single path segment and every read must begin with it.
+ 2. **Log in at boot, once.** `login_approle(role_id, secret_id)`. Both credentials are
+ retained in memory so the process can re-authenticate unattended; if that is
+ unacceptable in your threat model, call `logout()` after the last read and accept that
+ the process cannot recover from token expiry on its own.
+ 3. **Classify a login failure before reacting to it.** A rejected SecretID raises
+ `VaultCredentialExhausted` — Vault expires a SecretID by `secret_id_ttl` and by
+ `secret_id_num_uses`, so retrying cannot succeed and the orchestrator must issue a new
+ one. A 429 or 5xx raises `VaultTransportError`, which *is* worth a backed-off retry.
+ Never wrap login in an unbounded retry loop: a spent SecretID would spin forever.
+ 4. **Read secrets by path.** `get_secret("prod/binance/market-maker")` returns a
+ `SecretBundle`. Hand it to the exchange client with `bundle.as_dict()` — an explicit
+ call, so the plaintext never appears by accident.
+ 5. **Distinguish the three failure modes on a read.** `VaultPathViolation` is your own
+ bug (wrong environment, traversal, malformed path) and never reached the network.
+ `VaultSecretNotFound` means Vault answered 404 — which means the path is absent *or*
+ invisible to this policy *or* soft-deleted; check the policy before concluding the
+ secret is missing. `VaultPermissionDenied` means a freshly issued token was still
+ refused, i.e. the policy genuinely forbids the path.
+ 6. **Let the manager handle the token.** Each read checks the remaining TTL and, inside
+ `renew_margin`, renews via `auth/token/renew-self`. When renewal stops buying headroom
+ the token has hit its max TTL, which renewal cannot extend, so the manager
+ re-authenticates via AppRole exactly once. A 403 on a read likewise triggers exactly
+ one re-login before the error is raised — bounded, never a loop.
+ 7. **Bound staleness deliberately.** `cache_ttl` (default 300s) is the maximum time this
+ process can keep using a credential that has since been rotated. On a rotation
+ notification, call `invalidate(path)` rather than waiting out the TTL.
+ 8. **Decide the outage policy.** With `stale_if_error=True` (default) a Vault outage lets
+ the process keep trading on its last known credentials; with `False` a read raises
+ instead. Choose consciously — the safe answer differs for a market maker holding
+ inventory and for a batch job.
> Full procedure: see `references/workflows.md`.
> Standards reference: see `references/standards.md`.
> Printable pre-flight checklist: see `assets/checklist.md`.
## Common Pitfalls
- - **Using Root Tokens**: Passing the Vault Root Token into a trading bot. If the bot is compromised, the attacker has full control over all firm secrets.
- - **Leaking Secrets in Logs**: Accidentally printing `print(exchange.config)` or logging exceptions that contain the raw API key.
- - **Hammering Vault**: Requesting the API key from Vault for *every single order*. Vault will rate-limit you or crash. Retrieve the key once on boot and cache it in memory.
+ - **Treating the client's environment check as the security control.** A client-side
+ prefix test is defence in depth against a bad path string. If the AppRole's Vault policy
+ is broad, nothing in this module narrows it. Scope the policy; verify it with
+ `sys/capabilities-self`.
+ - **Enforcing the prefix with `startswith("prod/")`.** `"prod/../dev/binance"` passes that
+ test. The guard here splits into segments and rejects `.`, `..`, empty segments, and
+ anything outside a conservative character allowlist.
+ - **Reading the token's expiry as "the bot is authenticated forever".** AppRole tokens
+ carry a TTL and a max TTL, and renewal cannot extend past the max
+ (https://developer.hashicorp.com/vault/docs/concepts/tokens). A bot that logs in at boot
+ and never checks will take a 403 at an unpredictable moment — typically the first read
+ after a rotation, i.e. exactly when it needs to work.
+ - **Caching a secret with no expiry.** The original failure mode this module was written
+ against: security rotates and revokes an exchange key, the bot holds the old value in
+ memory indefinitely, and the first symptom is a wall of broker `401`s mid-session.
+ `cache_ttl` bounds it; `invalidate()` short-circuits it.
+ - **Reading interpretation into a 404.** Vault documents 404 as *"invalid path. This can
+ both mean that the path truly doesn't exist or that you don't have permission to view a
+ specific path"* (https://developer.hashicorp.com/vault/api-docs). Do not respond by
+ creating the secret — you may be papering over a policy gap. And a KV v2 path whose
+ latest version was soft-deleted also answers 404, with `data: null` and a
+ `deletion_time` in the metadata.
+ - **Retrying a rejected login.** A spent `secret_id_num_uses` or an expired
+ `secret_id_ttl` will never recover on retry. Distinguish it (`VaultCredentialExhausted`)
+ from a transport failure and escalate to the orchestrator instead of looping.
+ - **Shipping RoleID and SecretID together.** Injecting both as environment variables from
+ the same CI job collapses AppRole to a single shared password. HashiCorp's recommended
+ pattern delivers the SecretID response-wrapped, single-use, and ideally CIDR-bound.
+ - **Logging the config object.** `print(exchange.config)` and a traceback holding the
+ credential dict leak just as effectively as a hardcoded key. `SecretBundle` prints key
+ *names* only; `as_dict()` is the deliberate escape hatch.
+ - **Re-reading Vault on every order.** Vault Community Edition supports rate-limit
+ quotas, which answer `429` when exceeded
+ (https://developer.hashicorp.com/vault/docs/concepts/resource-quotas). Read at boot,
+ cache with a TTL, and refresh on rotation.
## Verification
- - Simulate the `VaultSecretsManager`. Initialize it using AppRole mock credentials, retrieve a secret, and ensure an exception is thrown if the bot tries to access a path outside its allowed environment.
- - Run `python scripts/test_vault_secrets_manager.py`.
+ - `python -m unittest discover -s skills/centralized-secrets-management-vault-integration/scripts`
+ runs the suite. It drives the manager through `InMemoryVaultTransport`, a deterministic
+ double that reproduces Vault's 404-for-invisible-paths behaviour, soft-deleted KV v2
+ versions, token TTL/max TTL, and single-use SecretIDs.
+ - Regression checks worth reading before trusting a change: traversal out of the
+ environment (`prod/../dev/...`), a rotated secret being picked up once `cache_ttl`
+ expires, re-login on max TTL, permanent failure on a spent SecretID, and `repr` of both
+ `SecretBundle` and `VaultSecretsManager` containing no secret material.
+ - Against a real Vault, confirm the policy — not the client — is the boundary: with the
+ bot's own token, attempt a read one environment across (`vault kv get
+ secret/dev/...` from a prod AppRole) and confirm Vault refuses it.
+ - Verify the audit device records the read, and that the recorded request contains the
+ path but no plaintext value.
## Related Skills
- - `api-key-least-privilege-audit-tool`
- `secrets-rotation-without-bot-downtime`
+ - `api-key-least-privilege-audit-tool`
+ - `sandbox-credential-leakage-prevention`
+ - `hardware-security-module-hsm-for-signing-keys`
+ - `structured-logging-for-post-incident-forensics`