Logging in Python

Logging is how a running program explains itself. Done well, logs make bugs reproducible, production issues diagnosable, and normal operation observable without scattering print() everywhere.

Why Logging Instead of print

Concern	print()	logging
Level control	no built-in way to silence selectively	levels let you filter noise
Destination	stdout only	files, streams, network, handlers
Formatting	manual	formatters centralize structure
Production safety	easy to leak noisy output	configurable per environment
Context	ad-hoc	easy to attach consistent metadata

Use print() for quick probes. Use logging when you want durable, filterable diagnostics.

Quick Reference

Level	Typical use
DEBUG	Detailed diagnostic info, internal state
INFO	Normal operational events — started, finished, key milestones
WARNING	Something unexpected but not a failure
ERROR	A problem that prevented part of the work
CRITICAL	Severe failure; the application may not continue

The Smallest Working Logger

import logging

logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)

def scale(values):
    logger.debug("scale input=%r", values)
    mean = sum(values) / len(values)
    logger.info("mean computed=%r", mean)
    return [v - mean for v in values]

__name__ gives each module its own logger name, which is the standard convention.

Log Levels in Practice

logger.debug("cache key=%s, hit=%s", key, hit)
logger.info("processing batch id=%s size=%d", batch_id, len(items))
logger.warning("fallback used, primary service unavailable")
logger.error("failed to write result: %s", exc_info=True)

• DEBUG is for details you usually do not want in production.
• INFO describes progress and meaningful milestones.
• WARNING is for recoverable oddities.
• ERROR signals a failure of some operation.
• CRITICAL is for fatal application-level problems.

Do not use INFO as a dump for everything. If every step is INFO, nothing stands out.

basicConfig

For simple programs and scripts, basicConfig is enough.

import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)-8s %(name)s %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S",
)

Call basicConfig once, early. It configures the root logger.

Logger, Handler, Formatter

The logging system has three layers:

1. Logger — the object you call (logger.info(...)).
2. Handler — where the log record goes (stream, file, etc.).
3. Formatter — how the record is rendered as text.

import logging

logger = logging.getLogger("analysis")
logger.setLevel(logging.DEBUG)

handler = logging.StreamHandler()
handler.setLevel(logging.INFO)

formatter = logging.Formatter(
    "%(asctime)s %(levelname)-8s %(name)s %(message)s"
)
handler.setFormatter(formatter)

logger.addHandler(handler)

Log records flow from logger to handler to formatter. A logger can have multiple handlers, each with its own level and format.

Common Handlers

Handler	Use for
StreamHandler	console / stdout or stderr
FileHandler	text file output
RotatingFileHandler	file with size-based rotation
TimedRotatingFileHandler	file with time-based rotation
NullHandler	silence logging (for library default)

from logging.handlers import RotatingFileHandler

file_handler = RotatingFileHandler(
    "app.log", maxBytes=10_000_000, backupCount=5
)
file_handler.setFormatter(logging.Formatter("%(asctime)s %(message)s"))
logger.addHandler(file_handler)

Configuration Styles

Programmatic

Good for applications where you want explicit control in code.

import logging

def setup_logging():
    logging.basicConfig(
        level=logging.INFO,
        format="%(asctime)s %(levelname)s %(name)s %(message)s",
    )

INI-style (fileConfig)

import logging.config

logging.config.fileConfig("logging.ini")

logging.ini:

[loggers]
keys=root,analysis

[handlers]
keys=console

[formatters]
keys=standard

[logger_root]
level=INFO
handlers=console

[logger_analysis]
level=DEBUG
handlers=console
qualname=analysis
propagate=0

[handler_console]
class=StreamHandler
level=INFO
formatter=standard
args=(sys.stdout,)

[formatter_standard]
format=%(asctime)s %(levelname)-8s %(name)s %(message)s
datefmt=%Y-%m-%d %H:%M:%S

DictConfig

For more complex setups, especially in web apps and services:

import logging.config

LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "standard": {
            "format": "%(asctime)s %(levelname)-8s %(name)s %(message)s"
        }
    },
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
            "level": "INFO",
            "formatter": "standard",
            "stream": "ext://sys.stdout",
        }
    },
    "loggers": {
        "analysis": {
            "level": "DEBUG",
            "handlers": ["console"],
            "propagate": False,
        }
    },
    "root": {
        "level": "INFO",
        "handlers": ["console"],
    },
}

logging.config.dictConfig(LOGGING)

Structured Logging

Pure text logs are hard to query and hard to parse reliably. Structured logging emits key-value pairs so downstream tools can filter and aggregate.

Simple structured style

logger.info(
    "batch complete",
    extra={"batch_id": batch_id, "rows": len(rows), "duration_s": duration}
)

Be cautious: extra requires the formatter and filters to handle unknown keys gracefully. Many projects use a structured logging library instead.

JSON logging

For machine-readable logs:

import json
import logging

class JsonFormatter(logging.Formatter):
    def format(self, record):
        payload = {
            "time": self.formatTime(record),
            "level": record.levelname,
            "name": record.name,
            "message": record.getMessage(),
        }
        if record.exc_info and record.exc_info[1]:
            payload["exception"] = str(record.exc_info[1])
        return json.dumps(payload)

handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
logger.addHandler(handler)

JSON logs are easier to index, filter, and ship to log aggregation systems.

Logging Exceptions

try:
    result = risky_operation()
except Exception as e:
    logger.error("risky_operation failed: %s", e, exc_info=True)
    result = fallback()

exc_info=True includes the traceback in the log record. This is the difference between "something failed" and "here is exactly why it failed."

Avoid bare except: that swallows exceptions silently.

Logging in Libraries vs Applications

• Libraries should use getLogger(__name__) and avoid configuring handlers. Let the application decide how logs are handled.
• Applications configure handlers, levels, and formatters.
• Library code should usually not call basicConfig.

A library's logger should default to NullHandler when no configuration is present, so it does not emit unexpected output.

Context and Correlation

For request-driven or batch processing systems, attaching context makes logs far more useful.

logger.info("starting", extra={"request_id": request_id, "user": user_id})

The exact mechanism depends on your framework. Many use logging filters or context variables to inject request-scoped metadata automatically.

Good context fields: request id, trace id, user id, job id, batch id, node, environment.

What to Log

Log for: - Startup and shutdown milestones. - Significant state transitions. - Recoverable problems and fallback behavior. - Failures with enough context to reproduce. - Key inputs and outputs at boundaries, when useful. - Performance-relevant operations when they matter.

Avoid logging: - Secrets, tokens, passwords, keys. - Full payloads when they are large or sensitive. - High-frequency per-item debug spam in production without a clear reason. - Personal data unless you have a specific, compliant reason.

Configuration by Environment

A common pattern:

Environment	Root level	App logger	Notes
Development	DEBUG	DEBUG	verbose, local console
Staging	INFO	INFO or DEBUG	more signal, less noise
Production	INFO or WARNING	INFO	ERROR/CRITICAL for alerts

Set the level via environment variable or configuration, not by editing code per environment.

import os
import logging

level = os.getenv("LOG_LEVEL", "INFO").upper()
logging.basicConfig(level=getattr(logging, level, logging.INFO))

Log Aggregation Basics

In larger systems, logs from many processes need to be searchable. Common goals: - Centralized collection (files, agents, sockets). - Structured format (JSON or key-value) for filtering. - Correlation via request/job ids. - Retention and rotation policies.

The logging module is the source; aggregation is the destination. Keep them decoupled.

Performance Notes

• Loggers are cheap when the level is not enabled, because the message is not formatted.
• Avoid expensive computation inside a log call if the level may be disabled.

# Avoid heavy work at DEBUG when DEBUG may be off
if logger.isEnabledFor(logging.DEBUG):
    logger.debug("stats=%s", expensive_stats())

• Use %-style formatting in the logging call, not f-strings, when the args may be skipped. The logging framework handles formatting only if the record is emitted.

Common Mistakes

• Logging everything at DEBUG in production — too much noise, too much cost.
• Swallowing exceptions silently — use ERROR with exc_info=True.
• Logging secrets — tokens, keys, passwords should never appear in logs.
• Using print for operational signals — you lose filtering and structure.
• Inconsistent logger names — __name__ is the easy convention.
• Configuring handlers in library code — leave that to the application.
• No correlation context — hard to trace one request across many log lines.
• Overusing CRITICAL — reserve it for truly fatal situations.

Practical Example — Logging a Data Pipeline Step

import logging
import time

logger = logging.getLogger("pipeline")

def clean_and_load(path, batch_id):
    logger.info("step=start", extra={"batch_id": batch_id, "path": path})
    t0 = time.time()

    try:
        rows = read_rows(path)
        logger.debug("loaded rows=%d", len(rows))
        cleaned = clean_numeric(rows)
        logger.info("cleaned rows=%d", len(cleaned))
    except FileNotFoundError:
        logger.error("file not found", exc_info=True)
        raise
    except Exception as e:
        logger.error("clean step failed", exc_info=True)
        raise

    duration = time.time() - t0
    logger.info(
        "step=complete",
        extra={"batch_id": batch_id, "duration_s": duration, "rows": len(cleaned)}
    )
    return cleaned

This pattern gives you: start/complete markers, row counts, error detail with tracebacks, and timing — all with consistent context.

Related Cheat Sheets

• python-testing — logging helps when tests need diagnostics
• pytest — configure log capture and assertion helpers
• debugging — logs and the debugger complement each other
• ml-preprocessing — logging transform steps helps diagnose pipeline issues
• model-monitoring — production observability builds on logging habits

Official Documentation

• logging
• logging.config
• logging.handlers

Last Updated

2026-09-06 — Initial version.