# Providers — Agent Instructions Each provider is an independent package with its own `pyproject.toml`, tests, and documentation. ## Structure - `provider.yaml` — metadata, dependencies, and configuration for the provider. - Building blocks: Hooks, Operators, Sensors, Transfers. - Use `version_compat.py` patterns for cross-version compatibility. ## Checklist - Keep `provider.yaml` metadata, docs, and tests in sync. - Don't upper-bound dependencies by default; add limits only with justification. - **Never edit an `apache-airflow-providers-*` `>=` floor in a `pyproject.toml`.** Only Release Managers do that; a contributor PR that raises one fails the `check_provider_dependency_bumps` selective check. When a provider needs something added to another provider in the same PR (most often `common.compat`), leave the existing version untouched and append the exact comment `# use next version` to that dependency line — release tooling rewrites it to the real version at release time: ```toml "apache-airflow-providers-common-compat>=1.8.0", # use next version ``` Modifying `common.compat` alongside any other provider makes the marker **mandatory** on that provider's common-compat line — a second selective check (`common_compat_changed_without_next_version`) enforces it. See [`contributing-docs/13_airflow_dependencies_and_extras.rst`](../contributing-docs/13_airflow_dependencies_and_extras.rst). - Tests live alongside the provider — mirror source paths in test directories. - Full guide: [`contributing-docs/12_provider_distributions.rst`](../contributing-docs/12_provider_distributions.rst) ## Changelog — never use newsfragments **Never create newsfragments for providers.** Providers are released from `main` in waves, so per-PR newsfragments are not consumed by the release process — the release manager regenerates the changelog from `git log`. The towncrier-managed `newsfragments/` workflow is used only by `airflow-core/`, `chart/`, and `dev/mypy/`. (`airflow-ctl/` follows the same "no newsfragments, direct edit" pattern as providers — see `dev/README_RELEASE_AIRFLOWCTL.md`.) When a provider change needs a user-visible note (typically a breaking change or important behavior change that warrants explanation), update the provider's `docs/changelog.rst` directly in the same PR, just below the `Changelog` header — exactly as the in-file `NOTE TO CONTRIBUTORS` block describes. Routine entries (features, bug fixes, misc) are collected automatically by the release manager from commit messages, so most PRs do not need to touch the changelog at all. ## Security: Connection extras must not be forwarded blindly to hooks/operators **Never pass the whole `Connection.extra` dict (or `**conn.extra_dejson`) as keyword arguments to a hook, operator, client constructor, or any underlying library call.** Forward only the specific extra keys you have explicitly reviewed and know are safe to expose. ### Why this matters — different security boundaries Airflow has two distinct user roles with different trust levels: - **Connection editors** — UI/API users with permission to create or edit Connections. They control the host, login, password, and the `extra` JSON blob. - **Dag authors** — users who write the Python code that constructs hooks and operators. They control which arguments are passed at call sites. These are *not* the same population, and the security model treats them differently. A Connection editor is trusted to supply credentials for a target system; they are **not** trusted to alter how the worker process behaves, load arbitrary Python code, change file paths the worker reads, or pass options into client libraries that the Dag author did not opt into. ### What goes wrong when extras are forwarded blindly Many client libraries accept constructor or method kwargs that are dangerous when attacker-controlled. Concrete examples seen in the wild: - A kwarg that names a Python callable, plugin path, or import string — attacker sets it to a module they control → **remote code execution** on the worker. - A kwarg that points at a local file (cert path, key file, log path, config file) — attacker redirects it to read or overwrite worker-side files. - A kwarg that sets a proxy, endpoint URL, or hostname — attacker redirects traffic to an MITM endpoint and harvests credentials or tokens for *other* systems. - A kwarg that toggles TLS verification, signing, or auth — attacker silently downgrades security. - A kwarg that controls subprocess execution, shell invocation, or template rendering — attacker reaches command/template injection. In all of these, the Connection editor effectively gains capabilities (RCE, file read/write, traffic redirection, auth bypass) that the security model does not grant them. ### The rule - **Allowlist, never passthrough.** In the hook, read each extra key by name (`conn.extra_dejson.get("region_name")`, `conn.extra_dejson.get("verify")`, …) and pass only those named values forward. Reject or ignore unknown keys. - **Do not** write `SomeClient(**conn.extra_dejson)`, `hook = MyHook(**conn.extra_dejson)`, or `kwargs.update(conn.extra_dejson)` followed by a downstream call. - When adding support for a new extra key, treat it like any other public argument: review what the underlying library does with it, and document it in the provider's connection docs. - If a Dag author genuinely needs to pass a non-allowlisted option, that option should be a **Dag-author-supplied argument** on the operator or hook (with its own review), not something a Connection editor can set. ### When reviewing provider PRs Flag any of these patterns: - `**conn.extra_dejson` or `**self.extra_dejson` spread into a constructor or call. - Looping over `conn.extra_dejson.items()` and forwarding every key. - New code paths where extras are merged into `kwargs` and then passed on unfiltered. - "Convenience" features that let users put arbitrary client kwargs into `extra` — they widen the Connection-editor blast radius.