etrade-oauth1-signature-flow · diff
v1.0.0 to v2.0.0
135 added, 23 removed. Audit A to A.
---
name: etrade-oauth1-signature-flow
- description: Use when integrating E*TRADE's API which uses OAuth1 (not OAuth2) signature-based
- authentication, handling request token acquisition, user authorization callback,
- access token exchange, and HMAC-SHA1 request signing for every API call.
+ description: Use when integrating E*TRADE's API, which uses OAuth 1.0a (not OAuth 2.0)
+ HMAC-SHA1 request signing — covering the three-legged request-token/authorize/access-token
+ flow with the mandatory oauth_callback=oob, RFC 5849 signature base string construction
+ including query parameters, and the 2-hour idle renewal and end-of-day token expiry rules.
domain: algorithmic-trading
subdomain: broker-integration
tags:
- broker-integration
- etrade
- oauth1
- hmac-sha1
- request-signing
+ - rfc-5849
brokers_frameworks:
- E*TRADE
- - OAuth1
+ - OAuth 1.0a (RFC 5849)
- HMAC-SHA1
- version: "1.0.0"
+ version: "2.0.0"
author: algo-trading-skills-contributors
license: Apache-2.0
---
## When to Use
Invoke this skill when building a trading bot for E\*TRADE. Unlike most modern broker APIs
- that use OAuth2 bearer tokens, E\*TRADE uses **OAuth1** with HMAC-SHA1 request signing.
- Every API request must include a signed `Authorization` header with nonce, timestamp, and
- signature. This skill covers the full OAuth1 flow and per-request signing.
+ that use OAuth 2.0 bearer tokens, E\*TRADE uses **OAuth 1.0a** with HMAC-SHA1 request
+ signing. Every API request must carry an `Authorization: OAuth ...` header with a nonce,
+ timestamp, and signature computed over a canonical *signature base string*. This skill
+ covers the full three-legged flow, per-request signing, and E\*TRADE's two distinct token
+ lifetime rules.
+ ## When NOT to Use
+
+ - **For any other broker.** OAuth 1.0a request signing is specific to E\*TRADE among the
+ brokers covered here. Schwab uses OAuth 2.0 PKCE (`schwab-api-oauth-pkce-flow`), Upstox
+ uses refresh-token rotation (`upstox-oauth-refresh-token-rotation`).
+ - **As a general OAuth 1.0a library.** The signing core here is RFC 5849-conformant, but
+ `RSA-SHA1` and `PLAINTEXT` are not implemented — E\*TRADE documents `HMAC-SHA1` as the
+ only supported `oauth_signature_method`.
+ - **As an HTTP client.** `scripts/etrade_auth.py` deliberately issues no requests and adds
+ no dependencies. It returns URLs and header values; you supply the transport, timeouts,
+ and retry policy.
+ - **As a headless/unattended login.** Leg two requires a human to approve in a browser and
+ copy back a verifier code. Automating a daily unattended restart is a separate problem —
+ see `headless-broker-auth-patterns` and `token-lifecycle-live-probing`.
+ - **As secret storage.** This holds credentials in memory only. Persistence, rotation, and
+ vaulting belong to `centralized-secrets-management-vault-integration` and
+ `secrets-rotation-without-bot-downtime`.
+
## Prerequisites
- - E\*TRADE developer account with consumer key and consumer secret.
- - Understanding of OAuth1 three-legged flow (request token → authorize → access token).
+ - E\*TRADE developer account with a consumer key and consumer secret. Sandbox and
+ production issue **separate** key pairs.
+ - A browser-capable human for leg two of the flow, once per calendar day.
+ - Host clock synchronized to within **5 minutes** of E\*TRADE's clock (`oauth_timestamp`
+ tolerance). See `clock-drift-monitoring-alerting-thresholds`.
## Workflow
- 1. **Request Token**: POST to E\*TRADE's request token endpoint with consumer key.
- 2. **User Authorization**: Redirect user to E\*TRADE authorization URL; receive verifier code.
- 3. **Access Token**: Exchange request token + verifier for access token/secret.
- 4. **Sign Requests**: Every API call signed with HMAC-SHA1 using consumer + access secrets.
- 5. **Token Refresh**: Access tokens expire daily; renew before market open.
+ 1. **Request Token** — `GET {base_url}/oauth/request_token`.
+ - All four token endpoints are **GET**, not POST.
+ - **`oauth_callback` is mandatory and must be `"oob"`**, whether or not the app has a
+ callback URL registered. It has no server-side default; omitting it fails the call.
+ `build_request_token_header()` sets it.
+ - Sign with the consumer secret and an **empty token secret** — the signing key still
+ ends in `&`.
+ - Parse the form-encoded response with `parse_token_response()`. An E\*TRADE error body
+ also parses cleanly as form data, so **decision point: reject any response without
+ both `oauth_token` and `oauth_token_secret` rather than storing empty strings** — an
+ empty-credential client fails opaquely at the first API call, not here.
+ 2. **User Authorization** — `GET https://us.etrade.com/e/t/etws/authorize?key=…&token=…`.
+ - **Decision point — this is a different host.** The authorize page is on
+ `us.etrade.com` for **both** sandbox and production. Deriving it from `api.etrade.com`
+ or `apisb.etrade.com` yields a dead URL.
+ - Percent-encode both query values. Request tokens routinely contain `+`, `/`, and `=`;
+ an unencoded `+` arrives at the server as a space and authorizes nothing.
+ - The user approves and receives an `oauth_verifier` on the Authorization Complete page
+ (or appended to a registered callback URL).
+ - **The request token is valid for 5 minutes.** If the user is slower, restart at leg
+ one — do not retry the exchange with the stale token.
+
+ 3. **Access Token** — `GET {base_url}/oauth/access_token`.
+ - Sign with the consumer secret **and the request token secret**, including
+ `oauth_token` (the request token) and `oauth_verifier`.
+
+ 4. **Sign Requests** — build the RFC 5849 base string, then HMAC-SHA1 it.
+ - Base string URI (§3.4.1.2): lowercase scheme and host, drop the default port, and
+ **exclude the query and fragment**.
+ - Parameters (§3.4.1.3.1): the `oauth_*` header parameters **plus the URL's query
+ parameters plus any form-encoded body parameters**. `oauth_signature` and `realm` are
+ excluded.
+ - Normalization (§3.4.1.3.2): percent-encode **first**, then sort by encoded name and,
+ for repeated names, by encoded value.
+ - Signing key: `percent_encode(consumer_secret) + "&" + percent_encode(token_secret)`.
+ - **Decision point — sign the exact URL you will send.** Signing a bare path and then
+ appending `?detailFlag=ALL` produces a signature E\*TRADE rejects, because the query
+ parameters were never in the base string.
+
+ 5. **Token Lifecycle** — two independent rules, with different remedies.
+ - **Idle inactivation:** after **2 hours** with no API request the access token is
+ inactivated. Remedy: `GET {base_url}/oauth/renew_access_token`
+ (`sign_renew_access_token()`).
+ - **Daily expiry:** the token expires at the **end of the current calendar day, US
+ Eastern**. Remedy: the full three-legged flow again, with a human. Renewal does not
+ extend past midnight ET.
+ - **Decision point — classify the failure before reacting.** A 401 after a quiet period
+ is an idle inactivation and needs a renew; a 401 after midnight ET needs
+ re-authorization. Looping renew calls against an expired token never recovers.
+ - Revoke on shutdown or suspected compromise: `GET {base_url}/oauth/revoke_access_token`
+ (`sign_revoke_access_token()`).
+
> Full procedure: see `references/workflows.md`.
- > Standards: see `references/standards.md`.
- > Checklist: see `assets/checklist.md`.
+ > Standards and endpoint table: see `references/standards.md`.
+ > Printable pre-flight checklist: see `assets/checklist.md`.
## Common Pitfalls
- - **Clock Skew in Signatures**: OAuth1 timestamps must be within 5 minutes of server time.
- - **Nonce Reuse**: Each request needs a unique nonce; reuse causes signature rejection.
- - **URL Encoding**: OAuth1 parameter encoding is strict (RFC 5849 percent-encoding).
+ - **Omitting query parameters from the signature base string.** The single most common
+ cause of a persistent 401 on E\*TRADE: `/v1/accounts/{id}/orders` signs fine, then
+ `?status=OPEN&count=50` is appended and every call is rejected. Query parameters are a
+ mandatory signature input (RFC 5849 §3.4.1.3.1), and the base string URI must carry no
+ query at all.
+ - **Sorting parameters before percent-encoding.** Raw `"a b" < "a+"` (0x20 < 0x2B), but
+ encoded `"a%20b" < "a%2B"` orders them the other way. The difference only surfaces on
+ values containing spaces, `+`, or non-ASCII — so it passes every simple test and fails on
+ the first symbol list or free-text order field.
+ - **Omitting `oauth_callback` on leg one.** E\*TRADE requires it on the request-token call
+ and documents that it must always be `"oob"`. There is no default.
+ - **Building the authorize URL from the API host.** The authorize page is on
+ `us.etrade.com`, not `api.etrade.com`/`apisb.etrade.com`, in both environments.
+ - **Not percent-encoding the request token in the authorize URL.** Base64-ish tokens
+ contain `+`, `/`, and `=`; an unencoded `+` is decoded server-side as a space.
+ - **Not percent-encoding the signature in the header.** Base64 output contains `+`, `/`,
+ and `=`, all of which must be escaped inside the quoted header value.
+ - **Renewing "before market open" and assuming you are covered.** The 2-hour idle clock is
+ independent of the trading session. A strategy that places one order at the open and then
+ goes quiet is inactivated by 11:30 ET even though the token has not expired.
+ - **Retrying a renew against a token that expired at midnight ET.** Renewal only reverses
+ idle inactivation. Past end-of-day US Eastern the token is gone and needs a human.
+ - **Reusing a nonce with the same timestamp.** E\*TRADE rejects it. Generate the nonce and
+ timestamp fresh per request; never cache a signed header for reuse.
+ - **Clock skew.** `oauth_timestamp` must be within 5 minutes of E\*TRADE's clock. A drifting
+ container clock produces signature rejections that look like credential failures.
+ - **Logging credential objects.** A default dataclass `repr` prints secrets in plain text
+ into logs and tracebacks. `OAuth1Credentials` and `OAuth1Token` mark secret fields
+ `repr=False`; keep it that way.
## Verification
- - Generate an OAuth1 signature and verify it matches expected HMAC-SHA1 output.
- - Simulate the full three-legged flow and verify token acquisition.
+ - Reproduce the **RFC 5849 §3.4.1.1** published base string with
+ `build_base_string("POST", "http://example.com/request?b5=%3D%253D&a3=a&c%40=&a2=r%20b", …)`
+ including the body parameters `c2=` and `a3=2 q` and a `realm` that must be dropped. This
+ vector exercises query-parameter inclusion, a repeated name, `realm` exclusion, and
+ encode-then-sort ordering at once.
+ - Reproduce the **X (Twitter) OAuth 1.0a documented signature** `Ls93hJiZbQ3akF3HF3x1Bz8/zU4=`
+ from its published base string and signing key, which pins HMAC-SHA1 and base64 end to end.
+ - Confirm `build_request_token_header()` emits `oauth_callback="oob"` and no `oauth_token`.
+ - Confirm `get_authorize_url()` starts with `https://us.etrade.com/e/t/etws/authorize?` for
+ both `use_sandbox=True` and `False`, and that a token `a+b/c=` encodes to `a%2Bb%2Fc%3D`.
+ - Negative checks: `parse_token_response()` must raise on an empty body, on
+ `oauth_problem=signature_invalid`, and on a response missing either token field;
+ `sign_request()` and `sign_renew_access_token()` must raise before an access token is set;
+ blank consumer credentials must raise at construction.
+ - Confirm `repr(OAuth1Credentials(...))` contains neither secret.
- Run `python scripts/test_etrade_auth.py` and confirm 100% pass rate.
## Related Skills
- `headless-broker-auth-patterns`
- `schwab-api-oauth-pkce-flow`
- `broker-agnostic-adapter-interface`
- ---
+ - `token-lifecycle-live-probing`
+ - `clock-drift-monitoring-alerting-thresholds`