Python Logging: Best Practices for Production (2026)

Published June 6, 2026 • 13 min read

Python's built-in logging module is powerful but full of footguns that trip up even experienced developers. Misconfigured logging shows up as duplicate log lines, missing context, secrets leaking into logs, or entire log streams disappearing silently. This guide covers the right way to set up logging for production Python services — from the basics of dictConfig and rotating file handlers through structured JSON logging with structlog, FastAPI request logging middleware, and correlation IDs that let you trace a request through every service.

Logging Module Architecture

Understanding the three components prevents most misconfigurations:

Loggers form a hierarchy: myapp.db is a child of myapp, which is a child of the root logger. By default, log records propagate up this hierarchy — which is why attaching a handler to the root logger affects every library in your process.

Production Setup with dictConfig

Always configure logging with dictConfig rather than calling basicConfig or attaching handlers manually. It gives you a declarative, version-controllable config that can be loaded from a YAML/JSON file:

# logging_config.py
import logging
import logging.config

LOGGING_CONFIG = {
    'version': 1,
    'disable_existing_loggers': False,   # IMPORTANT: keep third-party loggers alive
    'formatters': {
        'standard': {
            'format': '%(asctime)s [%(levelname)s] %(name)s: %(message)s',
            'datefmt': '%Y-%m-%dT%H:%M:%S',
        },
        'detailed': {
            'format': '%(asctime)s [%(levelname)s] %(name)s %(filename)s:%(lineno)d: %(message)s',
        },
    },
    'handlers': {
        'console': {
            'class'    : 'logging.StreamHandler',
            'level'    : 'INFO',
            'formatter': 'standard',
            'stream'   : 'ext://sys.stdout',  # stdout, not stderr
        },
        'file': {
            'class'       : 'logging.handlers.RotatingFileHandler',
            'level'       : 'DEBUG',
            'formatter'   : 'detailed',
            'filename'    : '/var/log/myapp/app.log',
            'maxBytes'    : 10 * 1024 * 1024,   # 10 MB
            'backupCount' : 5,
            'encoding'    : 'utf-8',
        },
        'error_file': {
            'class'       : 'logging.handlers.RotatingFileHandler',
            'level'       : 'ERROR',
            'formatter'   : 'detailed',
            'filename'    : '/var/log/myapp/errors.log',
            'maxBytes'    : 5 * 1024 * 1024,
            'backupCount' : 10,
            'encoding'    : 'utf-8',
        },
    },
    'loggers': {
        'myapp': {
            'level'    : 'DEBUG',
            'handlers' : ['console', 'file', 'error_file'],
            'propagate': False,   # stop here; don't also log to root
        },
        'sqlalchemy.engine': {
            'level'    : 'WARNING',  # suppress SQL echo in production
            'handlers' : ['console'],
            'propagate': False,
        },
        'uvicorn.access': {
            'level'    : 'INFO',
            'handlers' : ['console'],
            'propagate': False,
        },
    },
    'root': {
        'level'   : 'WARNING',
        'handlers': ['console'],
    },
}

def setup_logging():
    logging.config.dictConfig(LOGGING_CONFIG)

# In your app's startup:
# setup_logging()
# logger = logging.getLogger('myapp.api')

RotatingFileHandler vs TimedRotatingFileHandler

import logging
from logging.handlers import RotatingFileHandler, TimedRotatingFileHandler

# Size-based rotation — good for services with variable log volume
size_handler = RotatingFileHandler(
    filename    = 'app.log',
    maxBytes    = 10 * 1024 * 1024,  # 10 MB per file
    backupCount = 5,                  # keep 5 old files: app.log.1 ... app.log.5
    encoding    = 'utf-8',
)

# Time-based rotation — good for fixed-volume services, easier to correlate
# with deployment events
time_handler = TimedRotatingFileHandler(
    filename    = 'app.log',
    when        = 'midnight',    # 'midnight', 'h', 'w0' (Monday), etc.
    interval    = 1,
    backupCount = 30,            # keep 30 days of logs
    encoding    = 'utf-8',
    utc         = True,          # always rotate at UTC midnight
)

Structured Logging with structlog

structlog is a structured logging library that makes it trivial to add context to log events and output machine-readable JSON:

pip install structlog

# structlog_config.py
import logging
import structlog

def configure_structlog():
    shared_processors = [
        structlog.contextvars.merge_contextvars,      # inject context (request ID, user ID)
        structlog.stdlib.add_log_level,
        structlog.stdlib.add_logger_name,
        structlog.processors.TimeStamper(fmt='iso'),
        structlog.processors.StackInfoRenderer(),
        structlog.processors.format_exc_info,
    ]

    structlog.configure(
        processors=shared_processors + [
            structlog.stdlib.ProcessorFormatter.wrap_for_formatter,
        ],
        logger_factory=structlog.stdlib.LoggerFactory(),
        cache_logger_on_first_use=True,
    )

    formatter = structlog.stdlib.ProcessorFormatter(
        processor=structlog.processors.JSONRenderer(),
        foreign_pre_chain=shared_processors,
    )

    handler = logging.StreamHandler()
    handler.setFormatter(formatter)

    root_logger = logging.getLogger()
    root_logger.addHandler(handler)
    root_logger.setLevel(logging.INFO)

configure_structlog()

# Usage
logger = structlog.get_logger('myapp.api')
logger.info("request_completed",
            method="POST", path="/api/users",
            status_code=201, duration_ms=23.4, user_id=42)
# Output: {"method": "POST", "path": "/api/users", "status_code": 201,
#           "duration_ms": 23.4, "user_id": 42, "event": "request_completed",
#           "level": "info", "logger": "myapp.api", "timestamp": "2026-06-06T10:00:00Z"}

JSON Logging for ELK / Datadog

Pure stdlib JSON logging without structlog — useful when you need to minimize dependencies:

import json
import logging
import traceback
from datetime import datetime, timezone

class JSONFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str:
        log_entry = {
            'timestamp' : datetime.now(timezone.utc).isoformat(),
            'level'     : record.levelname,
            'logger'    : record.name,
            'message'   : record.getMessage(),
            'module'    : record.module,
            'function'  : record.funcName,
            'line'      : record.lineno,
        }
        # Attach any extra fields passed to the log call
        for key, value in record.__dict__.items():
            if key not in logging.LogRecord.__dict__ and not key.startswith('_'):
                log_entry[key] = value
        # Attach exception info
        if record.exc_info:
            log_entry['exception'] = self.formatException(record.exc_info)
        return json.dumps(log_entry, default=str)

# Setup
handler = logging.StreamHandler()
handler.setFormatter(JSONFormatter())
logging.getLogger('myapp').addHandler(handler)

# Usage — extra dict is spread into the JSON output
logger = logging.getLogger('myapp.service')
logger.info("user_created", extra={'user_id': 42, 'email': 'alice@example.com'})

FastAPI Request Logging Middleware

# middleware/logging.py
import time
import uuid
import logging
import structlog
from fastapi import Request, Response
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.types import ASGIApp

logger = structlog.get_logger('myapp.http')

class RequestLoggingMiddleware(BaseHTTPMiddleware):
    def __init__(self, app: ASGIApp, exclude_paths: set[str] | None = None):
        super().__init__(app)
        self.exclude_paths = exclude_paths or {'/health', '/metrics'}

    async def dispatch(self, request: Request, call_next) -> Response:
        if request.url.path in self.exclude_paths:
            return await call_next(request)

        request_id = request.headers.get('X-Request-ID', str(uuid.uuid4()))
        # Bind request context for all log calls within this request
        structlog.contextvars.clear_contextvars()
        structlog.contextvars.bind_contextvars(
            request_id = request_id,
            method     = request.method,
            path       = request.url.path,
            client_ip  = request.client.host if request.client else None,
        )

        start = time.perf_counter()
        logger.info("request_started")
        try:
            response = await call_next(request)
        except Exception:
            logger.exception("request_failed")
            raise
        finally:
            duration_ms = round((time.perf_counter() - start) * 1000, 2)
            logger.info("request_completed",
                        status_code=response.status_code,
                        duration_ms=duration_ms)
            structlog.contextvars.clear_contextvars()

        response.headers['X-Request-ID'] = request_id
        return response

# Register in FastAPI app
from fastapi import FastAPI
app = FastAPI()
app.add_middleware(RequestLoggingMiddleware, exclude_paths={'/health', '/ready', '/metrics'})

Correlation IDs with contextvars

Thread-local storage breaks with asyncio. Use contextvars.ContextVar instead to propagate a correlation ID through async call chains:

import contextvars
import logging
import uuid

# Module-level ContextVar — each asyncio task gets its own copy
_request_id: contextvars.ContextVar[str] = contextvars.ContextVar(
    'request_id', default='unknown'
)
_user_id: contextvars.ContextVar[int | None] = contextvars.ContextVar(
    'user_id', default=None
)

def set_request_context(request_id: str, user_id: int | None = None) -> None:
    _request_id.set(request_id)
    _user_id.set(user_id)

def get_request_id() -> str:
    return _request_id.get()

class CorrelationFilter(logging.Filter):
    """Inject correlation IDs into every log record."""
    def filter(self, record: logging.LogRecord) -> bool:
        record.request_id = _request_id.get()
        record.user_id    = _user_id.get()
        return True

# Add to handlers in dictConfig
# 'filters': {'correlation': {'()': 'myapp.logging_config.CorrelationFilter'}}
# handler['filters'] = ['correlation']

# Format string that includes the IDs:
# '%(asctime)s [%(levelname)s] [%(request_id)s] %(name)s: %(message)s'

Common Pitfalls

Using the root logger directly

Calling logging.info() writes to the root logger and affects every library in your process. Always use named loggers: logger = logging.getLogger(__name__). The __name__ pattern automatically creates a hierarchy matching your package structure.

Duplicate log entries

Caused by attaching a handler to both a child logger and the root logger, with propagate=True (the default). Each record travels up the hierarchy and gets handled twice. Fix: set propagate=False on any logger that has its own handlers, or remove the handler from the root logger.

Calling logging.basicConfig() in library code

basicConfig is a one-shot call — it only configures the root logger if it has no handlers yet. If a library calls it, it may silently hijack the application's logging setup. Libraries should only call logging.getLogger(__name__) and add a NullHandler: logging.getLogger(__name__).addHandler(logging.NullHandler()).

Logging inside a hot loop

Log writes involve I/O and formatting. Inside a tight loop processing millions of items, even logger.debug() calls that are below the handler's level still incur function-call overhead and string formatting. Use logger.isEnabledFor(logging.DEBUG) to guard expensive log statements.

Logging sensitive data

Never log passwords, tokens, credit card numbers, or PII. Use a logging.Filter to scrub or mask sensitive fields before they reach handlers. In structlog, add a processor that redacts known sensitive keys from the event dict.

Frequently Asked Questions

Should I use print() or logging?

Use logging for anything that needs to be observable in production — request events, errors, performance metrics, audit trails. Use print() only for quick scripts or interactive debugging that you'll remove before commit. The logging module provides levels, handlers, formatters, and filters that print() cannot match.

What logging level should I use in production?

Set INFO for the root logger and your application loggers in production. DEBUG generates too much volume and can expose sensitive data. Set WARNING or ERROR for noisy third-party libraries (SQLAlchemy, requests, boto3) that produce INFO-level output you don't need. Keep DEBUG in staging and development.

How do I log to multiple destinations (file + Datadog agent)?

Add multiple handlers to your logger in dictConfig. One StreamHandler outputs to stdout (picked up by the log shipper), and one RotatingFileHandler writes to disk as a backup. Avoid writing directly to Datadog's HTTP API from the logging handler — use the local Datadog Agent instead to avoid blocking the request thread on network I/O.

How do I suppress noisy third-party logs?

Add entries for the noisy logger in your dictConfig 'loggers' section with 'level': 'WARNING' and 'propagate': False. Common offenders: sqlalchemy.engine (SQL statements at INFO), boto3/botocore (HTTP requests), httpx, urllib3.

What is the difference between logger.exception() and logger.error()?

logger.exception(msg) is equivalent to logger.error(msg, exc_info=True) — it logs at ERROR level and automatically appends the current exception traceback. Always use logger.exception() inside an except block to capture the full stack trace. Use logger.error() for error conditions that don't involve an exception.