Python Packaging

Create, structure, and distribute Python packages using modern packaging tools, pyproject.toml, and publishing to PyPI.

When to Invoke

• Creating Python libraries for distribution

• Building CLI tools with entry points

• Publishing packages to PyPI or private repositories

• Setting up Python project structure (src layout vs flat)

• Configuring pyproject.toml, setup.py, or setup.cfg

• Building wheels and source distributions

• Versioning and releasing Python packages

• Creating namespace packages or multi-package projects

Core Concepts

Package Structure

• Source layout: src/package_name/ (recommended)

• Flat layout: package_name/ (simpler but less flexible)

• Package metadata: pyproject.toml, setup.py, or setup.cfg

• Distribution formats: wheel (.whl) and source distribution (.tar.gz)

Modern Packaging Standards

• PEP 517/518: Build system requirements

• PEP 621: Metadata in pyproject.toml

• PEP 660: Editable installs

• pyproject.toml: Single source of configuration

Build Backends

• setuptools: Traditional, widely used

• hatchling: Modern, opinionated

• flit: Lightweight, for pure Python

• poetry: Dependency management + packaging

Distribution

• PyPI: Python Package Index (public)

• TestPyPI: Testing before production

• Private repositories: JFrog, AWS CodeArtifact, etc.

Quick Start

Minimal Package Structure

my-package/ pyproject.toml README.md LICENSE src/ my_package/ init.py module.py tests/ test_module.py

Minimal pyproject.toml

[build-system] requires = ["setuptools>=61.0"] build-backend = "setuptools.build_meta"

[project] name = "my-package" version = "0.1.0" description = "A short description" authors = [{name = "Your Name", email = "you@example.com"}] readme = "README.md" requires-python = ">=3.8" dependencies = [ "requests>=2.28.0", ]

[project.optional-dependencies] dev = [ "pytest>=7.0", "black>=22.0", ]

Source Layout Configuration

[tool.setuptools.packages.find] where = ["src"]

Package Structure Patterns

Source Layout (Recommended)

my-package/ pyproject.toml README.md LICENSE .gitignore src/ my_package/ init.py core.py utils.py py.typed tests/ init.py test_core.py test_utils.py docs/ index.md

Advantages: prevents accidental imports from source, cleaner test imports, better isolation.

Flat Layout

my-package/ pyproject.toml README.md my_package/ init.py module.py tests/ test_module.py

Simpler, but can import package without installing.

Multi-Package Project

project/ pyproject.toml packages/ package-a/ src/ package_a/ package-b/ src/ package_b/ tests/

Dynamic Versioning

[build-system] requires = ["setuptools>=61.0", "setuptools-scm>=8.0"] build-backend = "setuptools.build_meta"

[project] name = "my-package" dynamic = ["version"]

[tool.setuptools.dynamic] version = {attr = "my_package.version"}

In init.py:

version = "1.0.0"

Or with setuptools-scm

from importlib.metadata import version version = version("my-package")

Building and Publishing

Build Package Locally

pip install build twine

Build distribution

python -m build

Check the distribution

twine check dist/*

Publishing to PyPI

Test on TestPyPI first

twine upload --repository testpypi dist/*

Install from TestPyPI to verify

pip install --index-url https://test.pypi.org/simple/ my-package

Publish to PyPI

twine upload dist/*

Using API tokens (recommended):

Create ~/.pypirc

[distutils] index-servers = pypi testpypi

[pypi] username = token password = pypi-...your-token...

[testpypi] username = token password = pypi-...your-test-token...

Testing Installation

Editable Install

pip install -e . pip install -e ".[dev]" pip install -e ".[dev,docs]"

Testing in Isolated Environment

python -m venv test-env source test-env/bin/activate

pip install dist/my_package-1.0.0-py3-none-any.whl python -c "import my_package; print(my_package.version)" my-tool --help

deactivate rm -rf test-env

Version Constraints

dependencies = [ "requests>=2.28.0,<3.0.0", # Compatible range "click~=8.1.0", # Compatible release (>=8.1.0,<8.2.0) "pydantic>=2.0", # Minimum version "numpy==1.24.3", # Exact version (avoid if possible) ]

Best Practices

• Use src/ layout for cleaner package structure

• Use pyproject.toml for modern packaging

• Pin build dependencies in build-system.requires

• Version appropriately with semantic versioning

• Include all metadata (classifiers, URLs, etc.)

• Test installation in clean environments

• Use TestPyPI before publishing to PyPI

• Document thoroughly with README and docstrings

• Include LICENSE file

• Automate publishing with CI/CD

Checklist for Publishing

• Code is tested (pytest passing)

• Documentation is complete (README, docstrings)

• Version number updated

• CHANGELOG.md updated

• License file included

• pyproject.toml is complete

• Package builds without errors

• Installation tested in clean environment

• CLI tools work (if applicable)

• PyPI metadata is correct (classifiers, keywords)

• GitHub repository linked

• Tested on TestPyPI first

• Git tag created for release

References

• references/packaging-guide.md
• full-featured pyproject.toml example with all tool configs, CLI patterns (Click and argparse), CI/CD publishing with GitHub Actions, multi-architecture wheels, data files, namespace packages, C extensions, git-based versioning, private package index, file templates (.gitignore, MANIFEST.in, README.md)

Resources

• Python Packaging Guide: https://packaging.python.org/

• PyPI: https://pypi.org/

• TestPyPI: https://test.pypi.org/

• setuptools documentation: https://setuptools.pypa.io/

• build: https://pypa-build.readthedocs.io/

• twine: https://twine.readthedocs.io/