Level: Intermediate to advanced
Suggested time: 5–7 hours
Python: 3.10+ (examples use standard-library features)
This week moves beyond writing functions and classes into shaping how they behave. You will learn how functions can retain state, how decorators wrap behavior, how Python manages resources and attribute access, and how classes can be customized at creation time. The lab brings these ideas together in a small reusable decorator library.
Learning goals
By the end, you should be able to:
- Explain closures and use decorators with parameters.
- Preserve a wrapped function's metadata with
functools. - Implement function-based and class-based decorators.
- Manage resources with context managers and
contextlib. - Explain descriptors and use them to validate attributes.
- Choose between dataclasses, protocols, abstract base classes, and class initialization hooks.
- Recognize when a metaclass is justified and when a simpler hook is clearer.
- Build and compose retry, caching, and timing decorators.
1. Closures: functions that remember
A closure is a function together with references to variables from its enclosing scope. The inner function can keep using those variables after the outer function returns.
def make_multiplier(factor):
def multiply(value):
return factor * value
return multiply
triple = make_multiplier(3)
assert triple(7) == 21multiply remembers factor. Python exposes captured variables through closure, though application code rarely needs to inspect it.
Closures are useful for small pieces of per-function state and are the foundation of many decorators. For mutable or rebinding state, use nonlocal:
def make_counter():
count = 0
def next_value():
nonlocal count
count += 1
return count
return next_value
counter = make_counter()
assert [counter(), counter(), counter()] == [1, 2, 3]Without nonlocal, assigning to count inside next_value would make it a local variable and reading it first would fail with UnboundLocalError.
Late binding in loops
Closures capture variables, not a frozen copy of their current value. A common surprise:
functions = []
for number in range(3):
functions.append(lambda: number)
assert [fn() for fn in functions] == [2, 2, 2]All three functions look up the same number after the loop. Bind the current value with a default argument:
functions = [lambda number=number: number for number in range(3)]
assert [fn() for fn in functions] == [0, 1, 2]2. Decorators and functools
A decorator accepts a callable and returns a callable, usually a wrapper that adds behavior before or after the original call.
from functools import wraps
from time import perf_counter
def timed(function):
@wraps(function)
def wrapper(*args, **kwargs):
started = perf_counter()
try:
return function(*args, **kwargs)
finally:
elapsed = perf_counter() - started
print(f"{function.__name__} took {elapsed:.6f}s")
return wrapper
@timed
def add(a, b):
"""Add two values."""
return a + b@timed is equivalent to add = timed(add). @wraps copies useful metadata such as the function name, docstring, annotations, and wrapped reference. It keeps tracebacks, help output, and introspection useful. Prefer it in almost every wrapper.
Use try/finally when cleanup or measurement must happen even if the wrapped function raises. The wrapper should normally return the wrapped function's result unchanged and let its exceptions propagate unless changing that behavior is intentional.
Decorators with arguments
A parameterized decorator adds an outer factory call:
from functools import wraps
def repeat(times):
def decorate(function):
@wraps(function)
def wrapper(*args, **kwargs):
result = None
for _ in range(times):
result = function(*args, **kwargs)
return result
return wrapper
return decorate
@repeat(3)
def greet(name):
print(f"Hello, {name}!")This is equivalent to greet = repeat(3)(greet). The three layers have distinct jobs: accept configuration, accept the target function, then handle calls.
A decorator that supports both @decorator and @decorator(option=...) needs to distinguish the two forms. Avoid adding that complexity unless callers need both spellings.
Useful functools tools
@functools.wraps(original)preserves the wrapped callable's metadata.functools.lru_cache(maxsize=128)memoizes calls using hashable arguments. It is appropriate for deterministic functions; it retains results and should not cache side effects or changing external data.functools.cacheis an unbounded cache (Python 3.9+); use it only when the key space and lifetime are controlled.functools.partial(function, fixed_arg=...)creates a callable with some arguments pre-filled.functools.singledispatchdispatches a function based on the type of its first argument.functools.cached_propertycomputes an instance property once and stores it on that instance (Python 3.8+).
Example:
from functools import lru_cache
@lru_cache(maxsize=256)
def fibonacci(n: int) -> int:
if n < 2:
return n
return fibonacci(n - 1) + fibonacci(n - 2)Cached arguments must be hashable. Be deliberate about cache invalidation and shared state.
3. Class-based decorators
A class can act as a decorator by accepting the target callable in init and implementing call:
from functools import update_wrapper
class CountCalls:
def __init__(self, function):
self.function = function
self.calls = 0
update_wrapper(self, function)
def __call__(self, *args, **kwargs):
self.calls += 1
return self.function(*args, **kwargs)
@CountCalls
def square(value):
return value * value
assert square(4) == 16
assert square.calls == 1update_wrapper serves the same metadata-preservation role as @wraps. A class-based decorator is handy when the decorator itself needs inspectable state or methods. For ordinary wrapping, a closure is often simpler.
Methods caveat: A callable decorator object is not automatically a descriptor. When a decorated function is stored on a class, method binding may not behave as expected. For a decorator intended to support instance methods, implement get (often by returning types.MethodType(self, instance)) or use a function wrapper. Include a method-binding example in tests when supporting methods is part of the contract.
4. Context managers and contextlib
A context manager brackets work with setup and cleanup. The with statement guarantees the exit protocol runs even if the body raises.
class ManagedFile:
def __init__(self, path, mode="r", encoding="utf-8"):
self.path = path
self.mode = mode
self.encoding = encoding
def __enter__(self):
self.file = open(self.path, self.mode, encoding=self.encoding)
return self.file
def __exit__(self, exc_type, exc_value, traceback):
self.file.close()
return False # Do not suppress exceptions.Prefer the standard library's file manager in real code: with open(path, encoding="utf-8") as file:. In custom exit, return a truthy value only when intentionally suppressing an exception.
contextlib provides concise building blocks:
from contextlib import contextmanager
from time import perf_counter
@contextmanager
def elapsed(label):
started = perf_counter()
try:
yield
finally:
print(f"{label}: {perf_counter() - started:.3f}s")
with elapsed("work"):
sum(range(1_000_000))The code before yield is enter logic; the code after it is exit logic. finally ensures cleanup happens on normal completion and exceptions.
Other useful tools include:
contextlib.suppress(SomeError)for a narrowly scoped exception that is safe to ignore.contextlib.ExitStackwhen the number of context managers is dynamic.contextlib.closing(obj)for an object withclose()but no context-manager protocol.contextlib.nullcontext(value)as a no-op context manager, often useful in conditional code.
5. Descriptors: controlling attribute access
A descriptor is an object defining one or more of get, set, or delete. When stored as a class attribute, Python calls these methods during attribute access. Properties are descriptors; custom descriptors can reuse validation behavior across many fields.
class Positive:
def __set_name__(self, owner, name):
self.name = name
self.storage_name = "_" + name
def __get__(self, instance, owner=None):
if instance is None:
return self
return getattr(instance, self.storage_name, 0)
def __set__(self, instance, value):
if value <= 0:
raise ValueError(f"{self.name} must be positive")
setattr(instance, self.storage_name, value)
class Product:
quantity = Positive()
def __init__(self, quantity):
self.quantity = quantity
item = Product(4)
assert item.quantity == 4__set_name__ is called when the owning class is created. A descriptor that defines set is a data descriptor and generally takes precedence over an instance attribute of the same name. A descriptor defining only get is a non-data descriptor and can be shadowed by an instance attribute.
Descriptors are powerful, but they add indirection. Use a property for one-off validation; use a descriptor when the same attribute behavior is shared across several classes or fields.
6. Choosing a class extension mechanism
Python offers several ways to customize class behavior. Pick the least powerful tool that expresses the requirement clearly.
__init_subclass__: a lightweight subclass hook
Python calls a base class's __init_subclass__ when a subclass is created. It can validate or register subclasses without introducing a metaclass.
class Plugin:
registry = {}
def __init_subclass__(cls, *, name=None, **kwargs):
super().__init_subclass__(**kwargs)
if name is not None:
if name in Plugin.registry:
raise ValueError(f"duplicate plugin name: {name}")
Plugin.registry[name] = cls
class JsonPlugin(Plugin, name="json"):
passForward unknown class keywords through super().__init_subclass__(**kwargs) so cooperative multiple inheritance can work.
Metaclasses: customizing class creation
A metaclass is the class of a class. It controls how class objects are built; the default is type. Metaclasses can inspect or transform a class namespace before the class exists.
class RequireDocstring(type):
def __new__(mcls, name, bases, namespace):
if name != "Base" and not namespace.get("__doc__"):
raise TypeError(f"{name} needs a docstring")
return super().__new__(mcls, name, bases, namespace)
class Base(metaclass=RequireDocstring):
"""Base for documented subclasses."""
class Concrete(Base):
"""A documented concrete type."""Metaclasses affect every subclass and can conflict when multiple bases have incompatible metaclasses. Use one when class creation itself must be controlled and __init_subclass__ or a class decorator cannot do the job. For simple registration, validation, or configuration, start with __init_subclass__.
Dataclasses: less boilerplate for data
dataclasses generates common methods from annotated fields:
from dataclasses import dataclass, field
@dataclass
class Job:
name: str
retries: int = 3
tags: list[str] = field(default_factory=list)
job = Job("thumbnail")
job.tags.append("image")Use default_factory for mutable defaults so each instance gets its own list. frozen=True makes attribute reassignment unavailable through the normal interface; it does not recursively freeze objects stored in fields. Dataclasses are a good fit for data containers, not a substitute for every class.
Protocols and ABCs: describing interfaces
A Protocol supports structural typing: an object is compatible if it has the required shape, whether or not it explicitly inherits from the protocol.
from typing import Protocol
class SupportsClose(Protocol):
def close(self) -> None: ...
def shut_down(resource: SupportsClose) -> None:
resource.close()An abstract base class (ABC) supports nominal relationships and can prevent instantiation until required methods are implemented:
from abc import ABC, abstractmethod
class Serializer(ABC):
@abstractmethod
def dumps(self, value: object) -> str:
"""Serialize a value."""
class JsonSerializer(Serializer):
def dumps(self, value: object) -> str:
import json
return json.dumps(value)Use a protocol when callers care about capabilities and third-party types should fit without inheritance. Use an ABC when a shared base implementation, explicit runtime family, or enforced subclass contract is useful. Static type checkers understand both; runtime checking of protocols requires @runtime_checkable and only verifies limited structural presence, not full method signatures.
7. Lab: a retry, caching, and timing decorator library
Build a small module named decorators.py. It should offer three configurable decorators:
@retry(attempts=3, delay=0.0, exceptions=(Exception,)): retry selected failures, then re-raise the final exception.@cache(maxsize=128): cache successful return values by arguments.@timed: measure each actual invocation and report elapsed time.
This implementation is intentionally small and synchronous. It does not provide async support, cache invalidation, jitter, or distributed coordination.
# decorators.py
from collections.abc import Callable
from functools import lru_cache, wraps
from time import perf_counter, sleep
from typing import ParamSpec, TypeVar
P = ParamSpec("P")
R = TypeVar("R")
def retry(
attempts: int = 3,
*,
delay: float = 0.0,
exceptions: tuple[type[Exception], ...] = (Exception,),
) -> Callable[[Callable[P, R]], Callable[P, R]]:
if attempts < 1:
raise ValueError("attempts must be at least 1")
if delay < 0:
raise ValueError("delay cannot be negative")
if not exceptions:
raise ValueError("exceptions cannot be empty")
def decorate(function: Callable[P, R]) -> Callable[P, R]:
@wraps(function)
def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
for attempt in range(1, attempts + 1):
try:
return function(*args, **kwargs)
except exceptions:
if attempt == attempts:
raise
if delay:
sleep(delay)
raise AssertionError("unreachable")
return wrapper
return decorate
def cache(maxsize: int = 128):
if maxsize < 1:
raise ValueError("maxsize must be at least 1")
def decorate(function):
cached = lru_cache(maxsize=maxsize)(function)
@wraps(function)
def wrapper(*args, **kwargs):
return cached(*args, **kwargs)
wrapper.cache_info = cached.cache_info
wrapper.cache_clear = cached.cache_clear
return wrapper
return decorate
def timed(function: Callable[P, R]) -> Callable[P, R]:
@wraps(function)
def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
started = perf_counter()
try:
return function(*args, **kwargs)
finally:
elapsed = perf_counter() - started
print(f"{function.__qualname__}: {elapsed:.6f}s")
return wrapperParamSpec carries a wrapped function's parameter types through the decorator, and TypeVar carries its return type. The runtime behavior works without static typing; these annotations help type checkers and editors.
Compose the decorators
from decorators import cache, retry, timed
calls = 0
@timed
@cache(maxsize=32)
@retry(attempts=3, delay=0.05, exceptions=(TimeoutError,))
def fetch_record(record_id: int) -> dict[str, int]:
global calls
calls += 1
if calls == 1:
raise TimeoutError("temporary failure")
return {"id": record_id}Decorators apply from the bottom up. Here, retry wraps the original function, cache wraps the retry wrapper, and timed wraps the cached callable. A repeated successful call with the same key is served from cache, so the outer timer measures the cache lookup on a hit. To time only actual computation, put @timed below @cache instead. To retry cache lookups as well, choose the order deliberately and document it.
The global above is included only to make the example easy to read; production code should usually keep state in an object or use a closure. Avoid caching mutable results if callers can mutate them, and remember that lru_cache requires hashable arguments. Exceptions are not cached by lru_cache.
Lab tasks
- Put the implementation in
decorators.pyand add a short module docstring describing the scope and limitations. - Demonstrate that retry calls again after a
TimeoutError, returns on success, and re-raises after the final attempt. - Demonstrate that
exceptions=(TimeoutError,)does not retry aValueError. - Call a cached function twice with the same hashable arguments; inspect
cache_info()and clear it withcache_clear(). - Verify that the timer reports elapsed time when the wrapped function returns and when it raises.
- Compose decorators in at least two orders and explain what the timer measures and what gets cached.
- Preserve
name,doc, and the signature metadata as far as possible withwraps. - Add a class-based
Retrydecorator as an extension. Decide explicitly whether it supports instance methods; if it does, implement and demonstrate descriptor binding.
Design questions
- Should attempts mean total calls or retries after the first call? This tutorial defines it as total calls.
- Should delays be fixed or exponential? This lab uses a fixed delay; a production implementation may add bounded exponential backoff and jitter.
- Should timing print, log, or call a supplied callback? Printing keeps the exercise self-contained; a library should usually accept a reporting hook or use logging.
- Should a cache be shared across decorated functions or scoped to one function? This implementation creates one cache per decorated function.
8. Suggested practice and review
Try these exercises after the lab:
- Write a closure that assigns monotonically increasing IDs and explain where its state lives.
- Write a decorator factory that rejects calls when a predicate is false, preserving metadata and exceptions.
- Implement a
Rangedescriptor that validates an inclusive lower and upper bound, then use it in two classes. - Use
ExitStackto open a variable number of files safely. - Register plugin subclasses with
__init_subclass__; then describe a requirement that would actually justify a metaclass. - Model a
Taskrecord with a dataclass and type a consumer against aProtocol.
Key takeaways
- A closure retains access to its enclosing scope;
nonlocalsupports rebinding captured state. - A decorator is callable transformation. Parameterized decorators add a factory layer.
wrapsandupdate_wrapperpreserve useful introspection metadata.- Context managers make setup and cleanup reliable, including on exceptions.
- Descriptors control class-level attribute behavior; properties cover many simple cases.
- Dataclasses model data, protocols describe structural capabilities, ABCs define nominal contracts, and
__init_subclass__handles many class-registration needs. - Metaclasses are a powerful class-creation hook and should be used only when simpler mechanisms cannot express the requirement.
- Decorator order changes semantics. Make it explicit and demonstrate it.
No comments:
Post a Comment