python-packaging · diff

git:20260307.47a5dbc to git:20260326.3e0cff6

1 added, 357 removed. Audit A to A.

---
name: python-packaging
description: Create distributable Python packages with proper project structure, setup.py/pyproject.toml, and publishing to PyPI. Use when packaging Python libraries, creating CLI tools, or distributing Python code.
---
# Python Packaging
Comprehensive guide to creating, structuring, and distributing Python packages using modern packaging tools, pyproject.toml, and publishing to PyPI.
## When to Use This Skill
- Creating Python libraries for distribution
- Building command-line tools with entry points
- Publishing packages to PyPI or private repositories
- Setting up Python project structure
- Creating installable packages with dependencies
- Building wheels and source distributions
- Versioning and releasing Python packages
- Creating namespace packages
- Implementing package metadata and classifiers
## Core Concepts
### 1. Package Structure
- **Source layout**: `src/package_name/` (recommended)
- **Flat layout**: `package_name/` (simpler but less flexible)
- **Package metadata**: pyproject.toml, setup.py, or setup.cfg
- **Distribution formats**: wheel (.whl) and source distribution (.tar.gz)
### 2. Modern Packaging Standards
- **PEP 517/518**: Build system requirements
- **PEP 621**: Metadata in pyproject.toml
- **PEP 660**: Editable installs
- **pyproject.toml**: Single source of configuration
### 3. Build Backends
- **setuptools**: Traditional, widely used
- **hatchling**: Modern, opinionated
- **flit**: Lightweight, for pure Python
- **poetry**: Dependency management + packaging
### 4. Distribution
- **PyPI**: Python Package Index (public)
- **TestPyPI**: Testing before production
- **Private repositories**: JFrog, AWS CodeArtifact, etc.
## Quick Start
### Minimal Package Structure
```
my-package/
├── pyproject.toml
├── README.md
├── LICENSE
├── src/
│ └── my_package/
│ ├── __init__.py
│ └── module.py
└── tests/
└── test_module.py
```
### Minimal pyproject.toml
```toml
[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"
[project]
name = "my-package"
version = "0.1.0"
description = "A short description"
authors = [{name = "Your Name", email = "you@example.com"}]
readme = "README.md"
requires-python = ">=3.8"
dependencies = [
"requests>=2.28.0",
]
[project.optional-dependencies]
dev = [
"pytest>=7.0",
"black>=22.0",
]
```
## Package Structure Patterns
### Pattern 1: Source Layout (Recommended)
```
my-package/
├── pyproject.toml
├── README.md
├── LICENSE
├── .gitignore
├── src/
│ └── my_package/
│ ├── __init__.py
│ ├── core.py
│ ├── utils.py
│ └── py.typed # For type hints
├── tests/
│ ├── __init__.py
│ ├── test_core.py
│ └── test_utils.py
└── docs/
└── index.md
```
**Advantages:**
- Prevents accidentally importing from source
- Cleaner test imports
- Better isolation
**pyproject.toml for source layout:**
```toml
[tool.setuptools.packages.find]
where = ["src"]
```
### Pattern 2: Flat Layout
```
my-package/
├── pyproject.toml
├── README.md
├── my_package/
│ ├── __init__.py
│ └── module.py
└── tests/
└── test_module.py
```
**Simpler but:**
- Can import package without installing
- Less professional for libraries
### Pattern 3: Multi-Package Project
```
project/
├── pyproject.toml
├── packages/
│ ├── package-a/
│ │ └── src/
│ │ └── package_a/
│ └── package-b/
│ └── src/
│ └── package_b/
└── tests/
```
## Complete pyproject.toml Examples
### Pattern 4: Full-Featured pyproject.toml
```toml
[build-system]
requires = ["setuptools>=61.0", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "my-awesome-package"
version = "1.0.0"
description = "An awesome Python package"
readme = "README.md"
requires-python = ">=3.8"
license = {text = "MIT"}
authors = [
{name = "Your Name", email = "you@example.com"},
]
maintainers = [
{name = "Maintainer Name", email = "maintainer@example.com"},
]
keywords = ["example", "package", "awesome"]
classifiers = [
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"License :: OSI Approved :: MIT License",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
]
dependencies = [
"requests>=2.28.0,<3.0.0",
"click>=8.0.0",
"pydantic>=2.0.0",
]
[project.optional-dependencies]
dev = [
"pytest>=7.0.0",
"pytest-cov>=4.0.0",
"black>=23.0.0",
"ruff>=0.1.0",
"mypy>=1.0.0",
]
docs = [
"sphinx>=5.0.0",
"sphinx-rtd-theme>=1.0.0",
]
all = [
"my-awesome-package[dev,docs]",
]
[project.urls]
Homepage = "https://github.com/username/my-awesome-package"
Documentation = "https://my-awesome-package.readthedocs.io"
Repository = "https://github.com/username/my-awesome-package"
"Bug Tracker" = "https://github.com/username/my-awesome-package/issues"
Changelog = "https://github.com/username/my-awesome-package/blob/main/CHANGELOG.md"
[project.scripts]
my-cli = "my_package.cli:main"
awesome-tool = "my_package.tools:run"
[project.entry-points."my_package.plugins"]
plugin1 = "my_package.plugins:plugin1"
[tool.setuptools]
package-dir = {"" = "src"}
zip-safe = false
[tool.setuptools.packages.find]
where = ["src"]
include = ["my_package*"]
exclude = ["tests*"]
[tool.setuptools.package-data]
my_package = ["py.typed", "*.pyi", "data/*.json"]
# Black configuration
[tool.black]
line-length = 100
target-version = ["py38", "py39", "py310", "py311"]
include = '\.pyi?$'
# Ruff configuration
[tool.ruff]
line-length = 100
target-version = "py38"
[tool.ruff.lint]
select = ["E", "F", "I", "N", "W", "UP"]
# MyPy configuration
[tool.mypy]
python_version = "3.8"
warn_return_any = true
warn_unused_configs = true
disallow_untyped_defs = true
# Pytest configuration
[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
addopts = "-v --cov=my_package --cov-report=term-missing"
# Coverage configuration
[tool.coverage.run]
source = ["src"]
omit = ["*/tests/*"]
[tool.coverage.report]
exclude_lines = [
"pragma: no cover",
"def __repr__",
"raise AssertionError",
"raise NotImplementedError",
]
```
### Pattern 5: Dynamic Versioning
```toml
[build-system]
requires = ["setuptools>=61.0", "setuptools-scm>=8.0"]
build-backend = "setuptools.build_meta"
[project]
name = "my-package"
dynamic = ["version"]
description = "Package with dynamic version"
[tool.setuptools.dynamic]
version = {attr = "my_package.__version__"}
# Or use setuptools-scm for git-based versioning
[tool.setuptools_scm]
write_to = "src/my_package/_version.py"
```
**In **init**.py:**
```python
# src/my_package/__init__.py
__version__ = "1.0.0"
# Or with setuptools-scm
from importlib.metadata import version
__version__ = version("my-package")
```
## Command-Line Interface (CLI) Patterns
### Pattern 6: CLI with Click
```python
# src/my_package/cli.py
import click
@click.group()
@click.version_option()
def cli():
"""My awesome CLI tool."""
pass
@cli.command()
@click.argument("name")
@click.option("--greeting", default="Hello", help="Greeting to use")
def greet(name: str, greeting: str):
"""Greet someone."""
click.echo(f"{greeting}, {name}!")
@cli.command()
@click.option("--count", default=1, help="Number of times to repeat")
def repeat(count: int):
"""Repeat a message."""
for i in range(count):
click.echo(f"Message {i + 1}")
def main():
"""Entry point for CLI."""
cli()
if __name__ == "__main__":
main()
```
**Register in pyproject.toml:**
```toml
[project.scripts]
my-tool = "my_package.cli:main"
```
**Usage:**
```bash
pip install -e .
my-tool greet World
my-tool greet Alice --greeting="Hi"
my-tool repeat --count=3
```
### Pattern 7: CLI with argparse
```python
# src/my_package/cli.py
import argparse
import sys
def main():
"""Main CLI entry point."""
parser = argparse.ArgumentParser(
description="My awesome tool",
prog="my-tool"
)
parser.add_argument(
"--version",
action="version",
version="%(prog)s 1.0.0"
)
subparsers = parser.add_subparsers(dest="command", help="Commands")
# Add subcommand
process_parser = subparsers.add_parser("process", help="Process data")
process_parser.add_argument("input_file", help="Input file path")
process_parser.add_argument(
"--output", "-o",
default="output.txt",
help="Output file path"
)
args = parser.parse_args()
if args.command == "process":
process_data(args.input_file, args.output)
else:
parser.print_help()
sys.exit(1)
def process_data(input_file: str, output_file: str):
"""Process data from input to output."""
print(f"Processing {input_file} -> {output_file}")
if __name__ == "__main__":
main()
```
## Building and Publishing
### Pattern 8: Build Package Locally
```bash
# Install build tools
pip install build twine
# Build distribution
python -m build
# This creates:
# dist/
# my-package-1.0.0.tar.gz (source distribution)
# my_package-1.0.0-py3-none-any.whl (wheel)
# Check the distribution
twine check dist/*
```
### Pattern 9: Publishing to PyPI
```bash
# Install publishing tools
pip install twine
# Test on TestPyPI first
twine upload --repository testpypi dist/*
# Install from TestPyPI to test
pip install --index-url https://test.pypi.org/simple/ my-package
# If all good, publish to PyPI
twine upload dist/*
```
**Using API tokens (recommended):**
```bash
# Create ~/.pypirc
[distutils]
index-servers =
pypi
testpypi
[pypi]
username = __token__
password = pypi-...your-token...
[testpypi]
username = __token__
password = pypi-...your-test-token...
```
### Pattern 10: Automated Publishing with GitHub Actions
```yaml
# .github/workflows/publish.yml
name: Publish to PyPI
on:
release:
types: [created]
jobs:
publish:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.11"
- name: Install dependencies
run: |
pip install build twine
- name: Build package
run: python -m build
- name: Check package
run: twine check dist/*
- name: Publish to PyPI
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
run: twine upload dist/*
```
- ## Advanced Patterns
-
- ### Pattern 11: Including Data Files
-
- ```toml
- [tool.setuptools.package-data]
- my_package = [
- "data/*.json",
- "templates/*.html",
- "static/css/*.css",
- "py.typed",
- ]
- ```
-
- **Accessing data files:**
-
- ```python
- # src/my_package/loader.py
- from importlib.resources import files
- import json
-
- def load_config():
- """Load configuration from package data."""
- config_file = files("my_package").joinpath("data/config.json")
- with config_file.open() as f:
- return json.load(f)
-
- # Python 3.9+
- from importlib.resources import files
-
- data = files("my_package").joinpath("data/file.txt").read_text()
- ```
-
- ### Pattern 12: Namespace Packages
-
- **For large projects split across multiple repositories:**
-
- ```
- # Package 1: company-core
- company/
- └── core/
- ├── __init__.py
- └── models.py
-
- # Package 2: company-api
- company/
- └── api/
- ├── __init__.py
- └── routes.py
- ```
-
- **Do NOT include **init**.py in the namespace directory (company/):**
-
- ```toml
- # company-core/pyproject.toml
- [project]
- name = "company-core"
-
- [tool.setuptools.packages.find]
- where = ["."]
- include = ["company.core*"]
-
- # company-api/pyproject.toml
- [project]
- name = "company-api"
-
- [tool.setuptools.packages.find]
- where = ["."]
- include = ["company.api*"]
- ```
-
- **Usage:**
-
- ```python
- # Both packages can be imported under same namespace
- from company.core import models
- from company.api import routes
- ```
-
- ### Pattern 13: C Extensions
-
- ```toml
- [build-system]
- requires = ["setuptools>=61.0", "wheel", "Cython>=0.29"]
- build-backend = "setuptools.build_meta"
-
- [tool.setuptools]
- ext-modules = [
- {name = "my_package.fast_module", sources = ["src/fast_module.c"]},
- ]
- ```
-
- **Or with setup.py:**
-
- ```python
- # setup.py
- from setuptools import setup, Extension
-
- setup(
- ext_modules=[
- Extension(
- "my_package.fast_module",
- sources=["src/fast_module.c"],
- include_dirs=["src/include"],
- )
- ]
- )
- ```
-
- ## Version Management
-
- ### Pattern 14: Semantic Versioning
-
- ```python
- # src/my_package/__init__.py
- __version__ = "1.2.3"
-
- # Semantic versioning: MAJOR.MINOR.PATCH
- # MAJOR: Breaking changes
- # MINOR: New features (backward compatible)
- # PATCH: Bug fixes
- ```
-
- **Version constraints in dependencies:**
-
- ```toml
- dependencies = [
- "requests>=2.28.0,<3.0.0", # Compatible range
- "click~=8.1.0", # Compatible release (~= 8.1.0 means >=8.1.0,<8.2.0)
- "pydantic>=2.0", # Minimum version
- "numpy==1.24.3", # Exact version (avoid if possible)
- ]
- ```
-
- ### Pattern 15: Git-Based Versioning
-
- ```toml
- [build-system]
- requires = ["setuptools>=61.0", "setuptools-scm>=8.0"]
- build-backend = "setuptools.build_meta"
-
- [project]
- name = "my-package"
- dynamic = ["version"]
-
- [tool.setuptools_scm]
- write_to = "src/my_package/_version.py"
- version_scheme = "post-release"
- local_scheme = "dirty-tag"
- ```
-
- **Creates versions like:**
-
- - `1.0.0` (from git tag)
- - `1.0.1.dev3+g1234567` (3 commits after tag)
-
- ## Testing Installation
-
- ### Pattern 16: Editable Install
-
- ```bash
- # Install in development mode
- pip install -e .
-
- # With optional dependencies
- pip install -e ".[dev]"
- pip install -e ".[dev,docs]"
-
- # Now changes to source code are immediately reflected
- ```
-
- ### Pattern 17: Testing in Isolated Environment
-
- ```bash
- # Create virtual environment
- python -m venv test-env
- source test-env/bin/activate # Linux/Mac
- # test-env\Scripts\activate # Windows
-
- # Install package
- pip install dist/my_package-1.0.0-py3-none-any.whl
-
- # Test it works
- python -c "import my_package; print(my_package.__version__)"
-
- # Test CLI
- my-tool --help
-
- # Cleanup
- deactivate
- rm -rf test-env
- ```
-
- ## Documentation
-
- ### Pattern 18: README.md Template
-
- ````markdown
- # My Package
-
- [![PyPI version](https://badge.fury.io/py/my-package.svg)](https://pypi.org/project/my-package/)
- [![Python versions](https://img.shields.io/pypi/pyversions/my-package.svg)](https://pypi.org/project/my-package/)
- [![Tests](https://github.com/username/my-package/workflows/Tests/badge.svg)](https://github.com/username/my-package/actions)
-
- Brief description of your package.
-
- ## Installation
-
- ```bash
- pip install my-package
- ```
- ````
-
- ## Quick Start
-
- ```python
- from my_package import something
-
- result = something.do_stuff()
- ```
-
- ## Features
-
- - Feature 1
- - Feature 2
- - Feature 3
-
- ## Documentation
-
- Full documentation: https://my-package.readthedocs.io
-
- ## Development
-
- ```bash
- git clone https://github.com/username/my-package.git
- cd my-package
- pip install -e ".[dev]"
- pytest
- ```
-
- ## License
-
- MIT
-
- ````
-
- ## Common Patterns
-
- ### Pattern 19: Multi-Architecture Wheels
-
- ```yaml
- # .github/workflows/wheels.yml
- name: Build wheels
-
- on: [push, pull_request]
-
- jobs:
- build_wheels:
- name: Build wheels on ${{ matrix.os }}
- runs-on: ${{ matrix.os }}
- strategy:
- matrix:
- os: [ubuntu-latest, windows-latest, macos-latest]
-
- steps:
- - uses: actions/checkout@v3
-
- - name: Build wheels
- uses: pypa/cibuildwheel@v2.16.2
-
- - uses: actions/upload-artifact@v3
- with:
- path: ./wheelhouse/*.whl
- ````
-
- ### Pattern 20: Private Package Index
-
- ```bash
- # Install from private index
- pip install my-package --index-url https://private.pypi.org/simple/
-
- # Or add to pip.conf
- [global]
- index-url = https://private.pypi.org/simple/
- extra-index-url = https://pypi.org/simple/
-
- # Upload to private index
- twine upload --repository-url https://private.pypi.org/ dist/*
- ```
-
- ## File Templates
-
- ### .gitignore for Python Packages
-
- ```gitignore
- # Build artifacts
- build/
- dist/
- *.egg-info/
- *.egg
- .eggs/
-
- # Python
- __pycache__/
- *.py[cod]
- *$py.class
- *.so
-
- # Virtual environments
- venv/
- env/
- ENV/
-
- # IDE
- .vscode/
- .idea/
- *.swp
-
- # Testing
- .pytest_cache/
- .coverage
- htmlcov/
-
- # Distribution
- *.whl
- *.tar.gz
- ```
-
- ### MANIFEST.in
-
- ```
- # MANIFEST.in
- include README.md
- include LICENSE
- include pyproject.toml
-
- recursive-include src/my_package/data *.json
- recursive-include src/my_package/templates *.html
- recursive-exclude * __pycache__
- recursive-exclude * *.py[co]
- ```
-
- ## Checklist for Publishing
-
- - [ ] Code is tested (pytest passing)
- - [ ] Documentation is complete (README, docstrings)
- - [ ] Version number updated
- - [ ] CHANGELOG.md updated
- - [ ] License file included
- - [ ] pyproject.toml is complete
- - [ ] Package builds without errors
- - [ ] Installation tested in clean environment
- - [ ] CLI tools work (if applicable)
- - [ ] PyPI metadata is correct (classifiers, keywords)
- - [ ] GitHub repository linked
- - [ ] Tested on TestPyPI first
- - [ ] Git tag created for release
+ For advanced patterns including data files, namespace packages, C extensions, version management, testing installation, documentation templates, and distribution workflows, see [references/advanced-patterns.md](references/advanced-patterns.md)