Python Packaging: Build, Publish and Install Libraries

Project Structure

Modern Python packages use the src layout — all importable code lives under a src/ directory, separate from project-level files like tests, docs, and configuration. The src layout prevents accidental imports of the uninstalled package during development and is the current recommendation from PyPA (Python Packaging Authority).

my-library/
├── src/
│   └── my_library/
│       ├── __init__.py
│       ├── core.py
│       └── utils.py
├── tests/
│   ├── conftest.py
│   └── test_core.py
├── docs/
├── pyproject.toml
├── README.md
├── LICENSE
└── .gitignore

# src/my_library/__init__.py
"""My Library — a reusable Python toolkit."""
from my_library.core import Client, Config

__version__ = "1.0.0"
__all__ = ["Client", "Config"]

pyproject.toml Deep Dive

pyproject.toml is the single source of truth for your package. It covers package metadata (name, version, description, classifiers), dependency requirements, build backend selection, and tool configuration for linters, formatters, and test runners. Every major packaging tool — pip, build, Poetry, Hatch, PDM — reads from this file.

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "my-library"
version = "1.0.0"
description = "A reusable Python toolkit"
readme = "README.md"
license = {file = "LICENSE"}
authors = [
    {name = "Alice Smith", email = "alice@example.com"}
]
keywords = ["python", "toolkit", "async"]
classifiers = [
    "Development Status :: 5 - Production/Stable",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.11",
    "Programming Language :: Python :: 3.12",
    "License :: OSI Approved :: MIT License",
    "Operating System :: OS Independent",
]
requires-python = ">=3.11"
dependencies = [
    "httpx>=0.27",
    "pydantic>=2.0",
]

[project.optional-dependencies]
redis = ["redis>=5.0"]
dev = ["pytest>=8.0", "ruff>=0.4"]

[project.urls]
Homepage = "https://example.com"
Repository = "https://github.com/alice/my-library"
Documentation = "https://docs.example.com"
"Bug Tracker" = "https://github.com/alice/my-library/issues"

[project.scripts]
my-lib = "my_library.cli:main"

[tool.hatch.build.targets.wheel]
packages = ["src/my_library"]

Build Backends

The build backend is the library that actually creates the distribution files. PEP 517 standardized the interface so that any frontend tool (pip, build) can work with any backend. The four main choices are Hatchling, Setuptools, Flit, and Poetry-core, each with different strengths. Hatchling is the current recommendation for new projects due to its speed and flexibility.

# Hatchling (recommended for new projects)
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

# Setuptools (most established, required for C extensions)
[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.build_meta"

# Flit (minimal, pure Python packages only)
[build-system]
requires = ["flit_core>=3.9"]
build-backend = "flit_core.buildapi"

# Poetry-core (used when managing with Poetry)
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

# For packages with C extensions, use setuptools + Cython
# setup.py (only needed when using C extensions)
from setuptools import setup, Extension
from Cython.Build import cythonize

extensions = [
    Extension("my_library._fast", ["src/my_library/_fast.pyx"])
]

setup(ext_modules=cythonize(extensions, compiler_directives={"language_level": "3"}))

Building Distributions

Python packages ship in two formats: a source distribution (sdist, a .tar.gz with your source code and build instructions) and a wheel (.whl, a pre-built ZIP archive that installs without any build step). Always provide both. pip prefers wheels for fast installation, while sdists are the authoritative source of truth and are required by PyPI.

# Install the build frontend
pip install build twine

# Build both sdist and wheel
python -m build
# Creates:
#   dist/my_library-1.0.0.tar.gz        (sdist)
#   dist/my_library-1.0.0-py3-none-any.whl  (pure Python wheel)

# Build only wheel (faster)
python -m build --wheel

# Build only sdist
python -m build --sdist

# Inspect wheel contents
unzip -l dist/my_library-1.0.0-py3-none-any.whl

# Validate the distribution files
twine check dist/*
# Checks README rendering, metadata validity, and common upload issues

# Install from local wheel (for testing before publish)
pip install dist/my_library-1.0.0-py3-none-any.whl

Versioning Strategies

Python packaging supports both static versioning (version string in pyproject.toml) and dynamic versioning (inferred from git tags at build time). Dynamic versioning via a plugin like hatch-vcs or setuptools-scm keeps the version as the single source of truth in your git tags, eliminating the need to update files before every release.

# Static versioning (simplest)
[project]
version = "1.2.3"

# Dynamic versioning from git tags with hatch-vcs
[build-system]
requires = ["hatchling", "hatch-vcs"]
build-backend = "hatchling.build"

[project]
dynamic = ["version"]

[tool.hatch.version]
source = "vcs"  # reads from git tags like v1.2.3

# Dynamic versioning with setuptools-scm
[build-system]
requires = ["setuptools>=68", "setuptools-scm"]
build-backend = "setuptools.build_meta"

[tool.setuptools_scm]
write_to = "src/my_library/_version.py"

# Release workflow with git tags (for dynamic versioning)
git tag -a v1.2.3 -m "Release 1.2.3"
git push origin v1.2.3

# The build will now produce my_library-1.2.3-*.whl
python -m build

# Semantic versioning (SemVer): MAJOR.MINOR.PATCH
# 1.0.0 → first stable release
# 1.1.0 → new backward-compatible feature
# 1.1.1 → bug fix
# 2.0.0 → breaking change

Publishing to PyPI

PyPI (the Python Package Index) is the official public registry for Python packages. Publishing requires a verified PyPI account and an API token. Twine is the standard upload tool — it validates packages before uploading and supports both PyPI and TestPyPI. Always test on TestPyPI first before publishing to the real index.

# Install twine
pip install twine

# Configure PyPI credentials (use API token, not password)
# Store in ~/.pypirc
cat > ~/.pypirc << 'EOF'
[distutils]
index-servers = pypi testpypi

[pypi]
repository = https://upload.pypi.org/legacy/
username = __token__
password = pypi-AgAAAA...your-api-token

[testpypi]
repository = https://test.pypi.org/legacy/
username = __token__
password = pypi-...testpypi-token
EOF

# Upload to TestPyPI first
twine upload --repository testpypi dist/*

# Install from TestPyPI to verify
pip install --index-url https://test.pypi.org/simple/ my-library

# Upload to production PyPI
twine upload dist/*

# Using environment variables (preferred for CI)
TWINE_USERNAME=__token__ TWINE_PASSWORD=pypi-... twine upload dist/*

Private Registries

Organizations often host private package registries to share internal libraries without publishing to PyPI. Popular solutions include AWS CodeArtifact, Nexus Repository, Artifactory, and DevPi. Configure pip and Poetry to install from private registries using index URLs and credentials stored securely in environment variables or the system keyring.

# Configure pip to use a private registry
pip config set global.extra-index-url https://nexus.company.com/repository/pypi/simple/

# Or per-install
pip install my-internal-lib \
  --extra-index-url https://nexus.company.com/repository/pypi/simple/ \
  --trusted-host nexus.company.com

# AWS CodeArtifact — get temporary token
export CODEARTIFACT_TOKEN=$(aws codeartifact get-authorization-token \
  --domain mycompany --domain-owner 123456789 \
  --query authorizationToken --output text)

pip install my-lib \
  --index-url https://aws:${CODEARTIFACT_TOKEN}@mycompany-123456789.d.codeartifact.us-east-1.amazonaws.com/pypi/my-repo/simple/

# Poetry private registry configuration
# pyproject.toml
[[tool.poetry.source]]
name = "company"
url = "https://nexus.company.com/repository/pypi/simple/"
priority = "supplemental"  # try PyPI first, fall back to this

# Configure credentials (stored in system keyring, not pyproject.toml)
# poetry config http-basic.company username password

Extras and Optional Dependencies

Package extras let users install optional feature sets with a single flag: pip install my-library[redis,async]. This pattern is common for libraries that have optional heavy dependencies — database drivers, cloud SDKs, visualization libraries — that most users don't need. Well-designed extras keep the base install lean while still providing a convenient one-command path to full functionality.

# pyproject.toml (standard packaging syntax)
[project.optional-dependencies]
redis = ["redis>=5.0", "hiredis>=2.0"]
postgres = ["asyncpg>=0.29"]
all-db = ["my-library[redis,postgres]"]  # meta-extra
dev = ["pytest>=8.0", "ruff>=0.4", "mypy>=1.10"]
docs = ["mkdocs>=1.6", "mkdocs-material>=9.5"]

# Install base package only
pip install my-library

# Install with redis extra
pip install "my-library[redis]"

# Install multiple extras
pip install "my-library[redis,postgres]"

# Install all extras
pip install "my-library[all-db]"

# Check extras from installed package
pip show my-library  # shows Requires-Dist for each extra

# In code — guard optional imports
try:
    import redis
    HAS_REDIS = True
except ImportError:
    HAS_REDIS = False

def get_cache():
    if not HAS_REDIS:
        raise ImportError("Install my-library[redis] to use caching")
    return redis.Redis()