Python Packaging

Comprehensive guide to creating, structuring, and distributing Python packages using modern packaging tools, pyproject.toml, and publishing to PyPI.

When to Use This Skill

• Creating Python libraries for distribution

• Building command-line tools with entry points

• Publishing packages to PyPI or private repositories

• Setting up Python project structure

• Creating installable packages with dependencies

• Building wheels and source distributions

• Versioning and releasing Python packages

• Creating namespace packages

• Implementing package metadata and classifiers

Core Concepts

1. Package Structure

• Source layout: src/package_name/ (recommended)

• Flat layout: package_name/ (simpler but less flexible)

• Package metadata: pyproject.toml, setup.py, or setup.cfg

• Distribution formats: wheel (.whl) and source distribution (.tar.gz)

2. Modern Packaging Standards

• PEP 517/518: Build system requirements

• PEP 621: Metadata in pyproject.toml

• PEP 660: Editable installs

• pyproject.toml: Single source of configuration

3. Build Backends

• setuptools: Traditional, widely used

• hatchling: Modern, opinionated

• flit: Lightweight, for pure Python

• poetry: Dependency management + packaging

4. Distribution

• PyPI: Python Package Index (public)

• TestPyPI: Testing before production

• Private repositories: JFrog, AWS CodeArtifact, etc.

Quick Start

Minimal Package Structure

my-package/ ├── pyproject.toml ├── README.md ├── LICENSE ├── src/ │ └── my_package/ │ ├── init.py │ └── module.py └── tests/ └── test_module.py

Minimal pyproject.toml

[build-system] requires = ["setuptools>=61.0"] build-backend = "setuptools.build_meta"

[project] name = "my-package" version = "0.1.0" description = "A short description" authors = [{name = "Your Name", email = "you@example.com"}] readme = "README.md" requires-python = ">=3.8" dependencies = [ "requests>=2.28.0", ]

[project.optional-dependencies] dev = [ "pytest>=7.0", "black>=22.0", ]

Package Structure Patterns

Pattern 1: Source Layout (Recommended)

my-package/ ├── pyproject.toml ├── README.md ├── LICENSE ├── .gitignore ├── src/ │ └── my_package/ │ ├── init.py │ ├── core.py │ ├── utils.py │ └── py.typed # For type hints ├── tests/ │ ├── init.py │ ├── test_core.py │ └── test_utils.py └── docs/ └── index.md

Advantages:

• Prevents accidentally importing from source

• Cleaner test imports

• Better isolation

pyproject.toml for source layout:

[tool.setuptools.packages.find] where = ["src"]

Pattern 2: Flat Layout

my-package/ ├── pyproject.toml ├── README.md ├── my_package/ │ ├── init.py │ └── module.py └── tests/ └── test_module.py

Simpler but:

• Can import package without installing

• Less professional for libraries

Pattern 3: Multi-Package Project

project/ ├── pyproject.toml ├── packages/ │ ├── package-a/ │ │ └── src/ │ │ └── package_a/ │ └── package-b/ │ └── src/ │ └── package_b/ └── tests/

Complete pyproject.toml Examples

Pattern 4: Full-Featured pyproject.toml

[build-system] requires = ["setuptools>=61.0", "wheel"] build-backend = "setuptools.build_meta"

[project] name = "my-awesome-package" version = "1.0.0" description = "An awesome Python package" readme = "README.md" requires-python = ">=3.8" license = {text = "MIT"} authors = [ {name = "Your Name", email = "you@example.com"}, ] maintainers = [ {name = "Maintainer Name", email = "maintainer@example.com"}, ] keywords = ["example", "package", "awesome"] classifiers = [ "Development Status :: 4 - Beta", "Intended Audience :: Developers", "License :: OSI Approved :: MIT License", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", ]

dependencies = [ "requests>=2.28.0,<3.0.0", "click>=8.0.0", "pydantic>=2.0.0", ]

[project.optional-dependencies] dev = [ "pytest>=7.0.0", "pytest-cov>=4.0.0", "black>=23.0.0", "ruff>=0.1.0", "mypy>=1.0.0", ] docs = [ "sphinx>=5.0.0", "sphinx-rtd-theme>=1.0.0", ] all = [ "my-awesome-package[dev,docs]", ]

[project.urls] Homepage = "https://github.com/username/my-awesome-package" Documentation = "https://my-awesome-package.readthedocs.io" Repository = "https://github.com/username/my-awesome-package" "Bug Tracker" = "https://github.com/username/my-awesome-package/issues" Changelog = "https://github.com/username/my-awesome-package/blob/main/CHANGELOG.md"

[project.scripts] my-cli = "my_package.cli:main" awesome-tool = "my_package.tools:run"

[project.entry-points."my_package.plugins"] plugin1 = "my_package.plugins:plugin1"

[tool.setuptools] package-dir = {"" = "src"} zip-safe = false

[tool.setuptools.packages.find] where = ["src"] include = ["my_package*"] exclude = ["tests*"]

[tool.setuptools.package-data] my_package = ["py.typed", ".pyi", "data/.json"]

Black configuration

[tool.black] line-length = 100 target-version = ["py38", "py39", "py310", "py311"] include = '.pyi?$'

Ruff configuration

[tool.ruff] line-length = 100 target-version = "py38"

[tool.ruff.lint] select = ["E", "F", "I", "N", "W", "UP"]

MyPy configuration

[tool.mypy] python_version = "3.8" warn_return_any = true warn_unused_configs = true disallow_untyped_defs = true

Pytest configuration

[tool.pytest.ini_options] testpaths = ["tests"] python_files = ["test_*.py"] addopts = "-v --cov=my_package --cov-report=term-missing"

Coverage configuration

[tool.coverage.run] source = ["src"] omit = ["/tests/"]

[tool.coverage.report] exclude_lines = [ "pragma: no cover", "def repr", "raise AssertionError", "raise NotImplementedError", ]

Pattern 5: Dynamic Versioning

[build-system] requires = ["setuptools>=61.0", "setuptools-scm>=8.0"] build-backend = "setuptools.build_meta"

[project] name = "my-package" dynamic = ["version"] description = "Package with dynamic version"

[tool.setuptools.dynamic] version = {attr = "my_package.version"}

Or use setuptools-scm for git-based versioning

[tool.setuptools_scm] write_to = "src/my_package/_version.py"

In init.py:

src/my_package/init.py

version = "1.0.0"

Or with setuptools-scm

from importlib.metadata import version version = version("my-package")

Command-Line Interface (CLI) Patterns

Pattern 6: CLI with Click

src/my_package/cli.py

import click

@click.group() @click.version_option() def cli(): """My awesome CLI tool.""" pass

@cli.command() @click.argument("name") @click.option("--greeting", default="Hello", help="Greeting to use") def greet(name: str, greeting: str): """Greet someone.""" click.echo(f"{greeting}, {name}!")

@cli.command() @click.option("--count", default=1, help="Number of times to repeat") def repeat(count: int): """Repeat a message.""" for i in range(count): click.echo(f"Message {i + 1}")

def main(): """Entry point for CLI.""" cli()

if name == "main": main()

Register in pyproject.toml:

[project.scripts] my-tool = "my_package.cli:main"

Usage:

pip install -e . my-tool greet World my-tool greet Alice --greeting="Hi" my-tool repeat --count=3

Pattern 7: CLI with argparse

src/my_package/cli.py

import argparse import sys

def main(): """Main CLI entry point.""" parser = argparse.ArgumentParser( description="My awesome tool", prog="my-tool" )

parser.add_argument(
    "--version",
    action="version",
    version="%(prog)s 1.0.0"
)

subparsers = parser.add_subparsers(dest="command", help="Commands")

# Add subcommand
process_parser = subparsers.add_parser("process", help="Process data")
process_parser.add_argument("input_file", help="Input file path")
process_parser.add_argument(
    "--output", "-o",
    default="output.txt",
    help="Output file path"
)

args = parser.parse_args()

if args.command == "process":
    process_data(args.input_file, args.output)
else:
    parser.print_help()
    sys.exit(1)

def process_data(input_file: str, output_file: str): """Process data from input to output.""" print(f"Processing {input_file} -> {output_file}")

if name == "main": main()

Building and Publishing

Pattern 8: Build Package Locally

Install build tools

pip install build twine

Build distribution

python -m build

This creates:

dist/

my-package-1.0.0.tar.gz (source distribution)

my_package-1.0.0-py3-none-any.whl (wheel)

Check the distribution

twine check dist/*

Pattern 9: Publishing to PyPI

Install publishing tools

pip install twine

Test on TestPyPI first

twine upload --repository testpypi dist/*

Install from TestPyPI to test

pip install --index-url https://test.pypi.org/simple/ my-package

If all good, publish to PyPI

twine upload dist/*

Using API tokens (recommended):

Create ~/.pypirc

[distutils] index-servers = pypi testpypi

[pypi] username = token password = pypi-...your-token...

[testpypi] username = token password = pypi-...your-test-token...

Pattern 10: Automated Publishing with GitHub Actions

.github/workflows/publish.yml

name: Publish to PyPI

on: release: types: [created]

jobs: publish: runs-on: ubuntu-latest

steps:
  - uses: actions/checkout@v3

  - name: Set up Python
    uses: actions/setup-python@v4
    with:
      python-version: "3.11"

  - name: Install dependencies
    run: |
      pip install build twine

  - name: Build package
    run: python -m build

  - name: Check package
    run: twine check dist/*

  - name: Publish to PyPI
    env:
      TWINE_USERNAME: __token__
      TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
    run: twine upload dist/*

For advanced patterns including data files, namespace packages, C extensions, version management, testing installation, documentation templates, and distribution workflows, see references/advanced-patterns.md