Monday, November 9, 2026

Advanced Functions and Object-Oriented Python

Level: Intermediate to advanced

Suggested time: 5–7 hours
Python: 3.10+ (examples use standard-library features)

This week moves beyond writing functions and classes into shaping how they behave. You will learn how functions can retain state, how decorators wrap behavior, how Python manages resources and attribute access, and how classes can be customized at creation time. The lab brings these ideas together in a small reusable decorator library.

Learning goals

By the end, you should be able to:

  • Explain closures and use decorators with parameters.
  • Preserve a wrapped function's metadata with functools.
  • Implement function-based and class-based decorators.
  • Manage resources with context managers and contextlib.
  • Explain descriptors and use them to validate attributes.
  • Choose between dataclasses, protocols, abstract base classes, and class initialization hooks.
  • Recognize when a metaclass is justified and when a simpler hook is clearer.
  • Build and compose retry, caching, and timing decorators.

1. Closures: functions that remember

A closure is a function together with references to variables from its enclosing scope. The inner function can keep using those variables after the outer function returns.

def make_multiplier(factor):
    def multiply(value):
        return factor * value
    return multiply

triple = make_multiplier(3)
assert triple(7) == 21

multiply remembers factor. Python exposes captured variables through closure, though application code rarely needs to inspect it.

Closures are useful for small pieces of per-function state and are the foundation of many decorators. For mutable or rebinding state, use nonlocal:

def make_counter():
    count = 0

    def next_value():
        nonlocal count
        count += 1
        return count

    return next_value

counter = make_counter()
assert [counter(), counter(), counter()] == [1, 2, 3]

Without nonlocal, assigning to count inside next_value would make it a local variable and reading it first would fail with UnboundLocalError.

Late binding in loops

Closures capture variables, not a frozen copy of their current value. A common surprise:

functions = []
for number in range(3):
    functions.append(lambda: number)

assert [fn() for fn in functions] == [2, 2, 2]

All three functions look up the same number after the loop. Bind the current value with a default argument:

functions = [lambda number=number: number for number in range(3)]
assert [fn() for fn in functions] == [0, 1, 2]

2. Decorators and functools

A decorator accepts a callable and returns a callable, usually a wrapper that adds behavior before or after the original call.

from functools import wraps
from time import perf_counter

def timed(function):
    @wraps(function)
    def wrapper(*args, **kwargs):
        started = perf_counter()
        try:
            return function(*args, **kwargs)
        finally:
            elapsed = perf_counter() - started
            print(f"{function.__name__} took {elapsed:.6f}s")
    return wrapper

@timed
def add(a, b):
    """Add two values."""
    return a + b

@timed is equivalent to add = timed(add). @wraps copies useful metadata such as the function name, docstring, annotations, and wrapped reference. It keeps tracebacks, help output, and introspection useful. Prefer it in almost every wrapper.

Use try/finally when cleanup or measurement must happen even if the wrapped function raises. The wrapper should normally return the wrapped function's result unchanged and let its exceptions propagate unless changing that behavior is intentional.

Decorators with arguments

A parameterized decorator adds an outer factory call:

from functools import wraps

def repeat(times):
    def decorate(function):
        @wraps(function)
        def wrapper(*args, **kwargs):
            result = None
            for _ in range(times):
                result = function(*args, **kwargs)
            return result
        return wrapper
    return decorate

@repeat(3)
def greet(name):
    print(f"Hello, {name}!")

This is equivalent to greet = repeat(3)(greet). The three layers have distinct jobs: accept configuration, accept the target function, then handle calls.

A decorator that supports both @decorator and @decorator(option=...) needs to distinguish the two forms. Avoid adding that complexity unless callers need both spellings.

Useful functools tools

  • @functools.wraps(original) preserves the wrapped callable's metadata.
  • functools.lru_cache(maxsize=128) memoizes calls using hashable arguments. It is appropriate for deterministic functions; it retains results and should not cache side effects or changing external data.
  • functools.cache is an unbounded cache (Python 3.9+); use it only when the key space and lifetime are controlled.
  • functools.partial(function, fixed_arg=...) creates a callable with some arguments pre-filled.
  • functools.singledispatch dispatches a function based on the type of its first argument.
  • functools.cached_property computes an instance property once and stores it on that instance (Python 3.8+).

Example:

from functools import lru_cache

@lru_cache(maxsize=256)
def fibonacci(n: int) -> int:
    if n < 2:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

Cached arguments must be hashable. Be deliberate about cache invalidation and shared state.

3. Class-based decorators

A class can act as a decorator by accepting the target callable in init and implementing call:

from functools import update_wrapper

class CountCalls:
    def __init__(self, function):
        self.function = function
        self.calls = 0
        update_wrapper(self, function)

    def __call__(self, *args, **kwargs):
        self.calls += 1
        return self.function(*args, **kwargs)

@CountCalls
def square(value):
    return value * value

assert square(4) == 16
assert square.calls == 1

update_wrapper serves the same metadata-preservation role as @wraps. A class-based decorator is handy when the decorator itself needs inspectable state or methods. For ordinary wrapping, a closure is often simpler.

Methods caveat: A callable decorator object is not automatically a descriptor. When a decorated function is stored on a class, method binding may not behave as expected. For a decorator intended to support instance methods, implement get (often by returning types.MethodType(self, instance)) or use a function wrapper. Include a method-binding example in tests when supporting methods is part of the contract.

4. Context managers and contextlib

A context manager brackets work with setup and cleanup. The with statement guarantees the exit protocol runs even if the body raises.

class ManagedFile:
    def __init__(self, path, mode="r", encoding="utf-8"):
        self.path = path
        self.mode = mode
        self.encoding = encoding

    def __enter__(self):
        self.file = open(self.path, self.mode, encoding=self.encoding)
        return self.file

    def __exit__(self, exc_type, exc_value, traceback):
        self.file.close()
        return False  # Do not suppress exceptions.

Prefer the standard library's file manager in real code: with open(path, encoding="utf-8") as file:. In custom exit, return a truthy value only when intentionally suppressing an exception.

contextlib provides concise building blocks:

from contextlib import contextmanager
from time import perf_counter

@contextmanager
def elapsed(label):
    started = perf_counter()
    try:
        yield
    finally:
        print(f"{label}: {perf_counter() - started:.3f}s")

with elapsed("work"):
    sum(range(1_000_000))

The code before yield is enter logic; the code after it is exit logic. finally ensures cleanup happens on normal completion and exceptions.

Other useful tools include:

  • contextlib.suppress(SomeError) for a narrowly scoped exception that is safe to ignore.
  • contextlib.ExitStack when the number of context managers is dynamic.
  • contextlib.closing(obj) for an object with close() but no context-manager protocol.
  • contextlib.nullcontext(value) as a no-op context manager, often useful in conditional code.

5. Descriptors: controlling attribute access

A descriptor is an object defining one or more of get, set, or delete. When stored as a class attribute, Python calls these methods during attribute access. Properties are descriptors; custom descriptors can reuse validation behavior across many fields.

class Positive:
    def __set_name__(self, owner, name):
        self.name = name
        self.storage_name = "_" + name

    def __get__(self, instance, owner=None):
        if instance is None:
            return self
        return getattr(instance, self.storage_name, 0)

    def __set__(self, instance, value):
        if value <= 0:
            raise ValueError(f"{self.name} must be positive")
        setattr(instance, self.storage_name, value)

class Product:
    quantity = Positive()

    def __init__(self, quantity):
        self.quantity = quantity

item = Product(4)
assert item.quantity == 4

__set_name__ is called when the owning class is created. A descriptor that defines set is a data descriptor and generally takes precedence over an instance attribute of the same name. A descriptor defining only get is a non-data descriptor and can be shadowed by an instance attribute.

Descriptors are powerful, but they add indirection. Use a property for one-off validation; use a descriptor when the same attribute behavior is shared across several classes or fields.

6. Choosing a class extension mechanism

Python offers several ways to customize class behavior. Pick the least powerful tool that expresses the requirement clearly.

__init_subclass__: a lightweight subclass hook

Python calls a base class's __init_subclass__ when a subclass is created. It can validate or register subclasses without introducing a metaclass.

class Plugin:
    registry = {}

    def __init_subclass__(cls, *, name=None, **kwargs):
        super().__init_subclass__(**kwargs)
        if name is not None:
            if name in Plugin.registry:
                raise ValueError(f"duplicate plugin name: {name}")
            Plugin.registry[name] = cls

class JsonPlugin(Plugin, name="json"):
    pass

Forward unknown class keywords through super().__init_subclass__(**kwargs) so cooperative multiple inheritance can work.

Metaclasses: customizing class creation

A metaclass is the class of a class. It controls how class objects are built; the default is type. Metaclasses can inspect or transform a class namespace before the class exists.

class RequireDocstring(type):
    def __new__(mcls, name, bases, namespace):
        if name != "Base" and not namespace.get("__doc__"):
            raise TypeError(f"{name} needs a docstring")
        return super().__new__(mcls, name, bases, namespace)

class Base(metaclass=RequireDocstring):
    """Base for documented subclasses."""

class Concrete(Base):
    """A documented concrete type."""

Metaclasses affect every subclass and can conflict when multiple bases have incompatible metaclasses. Use one when class creation itself must be controlled and __init_subclass__ or a class decorator cannot do the job. For simple registration, validation, or configuration, start with __init_subclass__.

Dataclasses: less boilerplate for data

dataclasses generates common methods from annotated fields:

from dataclasses import dataclass, field

@dataclass
class Job:
    name: str
    retries: int = 3
    tags: list[str] = field(default_factory=list)

job = Job("thumbnail")
job.tags.append("image")

Use default_factory for mutable defaults so each instance gets its own list. frozen=True makes attribute reassignment unavailable through the normal interface; it does not recursively freeze objects stored in fields. Dataclasses are a good fit for data containers, not a substitute for every class.

Protocols and ABCs: describing interfaces

A Protocol supports structural typing: an object is compatible if it has the required shape, whether or not it explicitly inherits from the protocol.

from typing import Protocol

class SupportsClose(Protocol):
    def close(self) -> None: ...

def shut_down(resource: SupportsClose) -> None:
    resource.close()

An abstract base class (ABC) supports nominal relationships and can prevent instantiation until required methods are implemented:

from abc import ABC, abstractmethod

class Serializer(ABC):
    @abstractmethod
    def dumps(self, value: object) -> str:
        """Serialize a value."""

class JsonSerializer(Serializer):
    def dumps(self, value: object) -> str:
        import json
        return json.dumps(value)

Use a protocol when callers care about capabilities and third-party types should fit without inheritance. Use an ABC when a shared base implementation, explicit runtime family, or enforced subclass contract is useful. Static type checkers understand both; runtime checking of protocols requires @runtime_checkable and only verifies limited structural presence, not full method signatures.

7. Lab: a retry, caching, and timing decorator library

Build a small module named decorators.py. It should offer three configurable decorators:

  • @retry(attempts=3, delay=0.0, exceptions=(Exception,)): retry selected failures, then re-raise the final exception.
  • @cache(maxsize=128): cache successful return values by arguments.
  • @timed: measure each actual invocation and report elapsed time.

This implementation is intentionally small and synchronous. It does not provide async support, cache invalidation, jitter, or distributed coordination.

# decorators.py
from collections.abc import Callable
from functools import lru_cache, wraps
from time import perf_counter, sleep
from typing import ParamSpec, TypeVar

P = ParamSpec("P")
R = TypeVar("R")


def retry(
    attempts: int = 3,
    *,
    delay: float = 0.0,
    exceptions: tuple[type[Exception], ...] = (Exception,),
) -> Callable[[Callable[P, R]], Callable[P, R]]:
    if attempts < 1:
        raise ValueError("attempts must be at least 1")
    if delay < 0:
        raise ValueError("delay cannot be negative")
    if not exceptions:
        raise ValueError("exceptions cannot be empty")

    def decorate(function: Callable[P, R]) -> Callable[P, R]:
        @wraps(function)
        def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
            for attempt in range(1, attempts + 1):
                try:
                    return function(*args, **kwargs)
                except exceptions:
                    if attempt == attempts:
                        raise
                    if delay:
                        sleep(delay)
            raise AssertionError("unreachable")
        return wrapper
    return decorate


def cache(maxsize: int = 128):
    if maxsize < 1:
        raise ValueError("maxsize must be at least 1")

    def decorate(function):
        cached = lru_cache(maxsize=maxsize)(function)
        @wraps(function)
        def wrapper(*args, **kwargs):
            return cached(*args, **kwargs)
        wrapper.cache_info = cached.cache_info
        wrapper.cache_clear = cached.cache_clear
        return wrapper
    return decorate


def timed(function: Callable[P, R]) -> Callable[P, R]:
    @wraps(function)
    def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
        started = perf_counter()
        try:
            return function(*args, **kwargs)
        finally:
            elapsed = perf_counter() - started
            print(f"{function.__qualname__}: {elapsed:.6f}s")
    return wrapper

ParamSpec carries a wrapped function's parameter types through the decorator, and TypeVar carries its return type. The runtime behavior works without static typing; these annotations help type checkers and editors.

Compose the decorators

from decorators import cache, retry, timed

calls = 0

@timed
@cache(maxsize=32)
@retry(attempts=3, delay=0.05, exceptions=(TimeoutError,))
def fetch_record(record_id: int) -> dict[str, int]:
    global calls
    calls += 1
    if calls == 1:
        raise TimeoutError("temporary failure")
    return {"id": record_id}

Decorators apply from the bottom up. Here, retry wraps the original function, cache wraps the retry wrapper, and timed wraps the cached callable. A repeated successful call with the same key is served from cache, so the outer timer measures the cache lookup on a hit. To time only actual computation, put @timed below @cache instead. To retry cache lookups as well, choose the order deliberately and document it.

The global above is included only to make the example easy to read; production code should usually keep state in an object or use a closure. Avoid caching mutable results if callers can mutate them, and remember that lru_cache requires hashable arguments. Exceptions are not cached by lru_cache.

Lab tasks

  1. Put the implementation in decorators.py and add a short module docstring describing the scope and limitations.
  2. Demonstrate that retry calls again after a TimeoutError, returns on success, and re-raises after the final attempt.
  3. Demonstrate that exceptions=(TimeoutError,) does not retry a ValueError.
  4. Call a cached function twice with the same hashable arguments; inspect cache_info() and clear it with cache_clear().
  5. Verify that the timer reports elapsed time when the wrapped function returns and when it raises.
  6. Compose decorators in at least two orders and explain what the timer measures and what gets cached.
  7. Preserve name, doc, and the signature metadata as far as possible with wraps.
  8. Add a class-based Retry decorator as an extension. Decide explicitly whether it supports instance methods; if it does, implement and demonstrate descriptor binding.

Design questions

  • Should attempts mean total calls or retries after the first call? This tutorial defines it as total calls.
  • Should delays be fixed or exponential? This lab uses a fixed delay; a production implementation may add bounded exponential backoff and jitter.
  • Should timing print, log, or call a supplied callback? Printing keeps the exercise self-contained; a library should usually accept a reporting hook or use logging.
  • Should a cache be shared across decorated functions or scoped to one function? This implementation creates one cache per decorated function.

8. Suggested practice and review

Try these exercises after the lab:

  1. Write a closure that assigns monotonically increasing IDs and explain where its state lives.
  2. Write a decorator factory that rejects calls when a predicate is false, preserving metadata and exceptions.
  3. Implement a Range descriptor that validates an inclusive lower and upper bound, then use it in two classes.
  4. Use ExitStack to open a variable number of files safely.
  5. Register plugin subclasses with __init_subclass__; then describe a requirement that would actually justify a metaclass.
  6. Model a Task record with a dataclass and type a consumer against a Protocol.

Key takeaways

  • A closure retains access to its enclosing scope; nonlocal supports rebinding captured state.
  • A decorator is callable transformation. Parameterized decorators add a factory layer.
  • wraps and update_wrapper preserve useful introspection metadata.
  • Context managers make setup and cleanup reliable, including on exceptions.
  • Descriptors control class-level attribute behavior; properties cover many simple cases.
  • Dataclasses model data, protocols describe structural capabilities, ABCs define nominal contracts, and __init_subclass__ handles many class-registration needs.
  • Metaclasses are a powerful class-creation hook and should be used only when simpler mechanisms cannot express the requirement.
  • Decorator order changes semantics. Make it explicit and demonstrate it.

No comments:

Post a Comment

Advanced Functions and Object-Oriented Python

Level: Intermediate to advanced Suggested time: 5–7 hours Python: 3.10+ (examples use standard-library features) This week moves beyond w...