Overview
By the end of this tutorial, you will be able to write decorators and context managers confidently, control attribute access with descriptors, customize class creation, and choose the right tool for modeling data and interfaces.
This course walks you through: functions and classes are objects (names bound to them), and dunder methods are the hooks Python calls for you. Every example runs on Python 3.11 or newer; outputs shown were produced on Python 3.13.
Part | Topic | You will be able to |
|---|---|---|
1 | Closures | Explain how a function keeps variables alive, and avoid the late-binding trap |
2 | Decorators and | Write plain, parameterized, and class-based decorators that preserve metadata |
3 | Context managers | Write them as classes and with |
4 | Descriptors | Build reusable validated attributes and explain how |
5 | Metaclasses and | Explain how a class is created, and build a plugin registry without a metaclass |
6 | Dataclasses | Declare data classes with defaults, ordering, immutability, and slots |
7 |
| Pick between nominal and structural typing for an interface |
Lab | Decorator library | Ship a tested |
Prerequisites: Comfort with functions, classes, and exceptions.
Time: about 5 hours: roughly 2 hours of reading with a REPL open, and 3 hours for the lab and exercises.
1. Closures
A closure is a function that remembers variables from the scope where it was defined, even after that scope has finished running.
Functions are objects, so a function can be created inside another function and returned. The inner function keeps references to the outer variables it uses; those are called free variables.
def make_multiplier(n):
def multiply(x):
return x * n # n is a free variable, taken from make_multiplier
return multiply
double = make_multiplier(2)
triple = make_multiplier(3)
print(double(5), triple(5)) # 10 15Each call to make_multiplier creates a fresh n and a fresh multiply, so double and triple do not interfere.
Looking inside a closure
The remembered variables live in cells, which you can inspect:
def make_counter():
count = 0
def counter():
nonlocal count
count += 1
return count
return counter
c1, c2 = make_counter(), make_counter()
print(c1(), c1(), c2()) # 1 2 1: each closure has its own count
print(c1.__code__.co_freevars) # ('count',)
print(c1.__closure__[0].cell_contents) # 2A closure is a lightweight alternative to a class with one method and a bit of state.
nonlocal: rebinding a free variable
Reading a free variable just works. Assigning to a name inside a function makes it local to that function, so rebinding an outer variable needs nonlocal.
def broken():
total = 0
def add(x):
total += x # total is treated as local here, and has no value yet
return add
broken()(1)
# UnboundLocalError: cannot access local variable 'total' where it is not associated with a valueAdding nonlocal total inside add fixes it. Mutating an outer object (such as items.append(x)) never needs nonlocal, because it does not rebind the name; this is the mutate-versus-rebind distinction.
The late-binding trap
A closure looks up its free variables when it is called, not when it is defined. In a loop, every closure then sees the variable's final value.
funcs = [lambda: i for i in range(3)]
print([f() for f in funcs]) # [2, 2, 2]: all three share one iThree ways to capture the current value:
funcs = [lambda i=i: i for i in range(3)] # default argument is evaluated now
print([f() for f in funcs]) # [0, 1, 2]
from functools import partial
funcs = [partial(lambda i: i, i) for i in range(3)] # partial stores the argument now
print([f() for f in funcs]) # [0, 1, 2]
def make(i): # a factory function gets its own scope
return lambda: i
funcs = [make(i) for i in range(3)] # [0, 1, 2]Closures are the mechanism behind decorators, which come next: a decorator is a function that returns a closure around the function it received.
2. Decorators and functools
A decorator is a function that takes a function and returns a replacement for it, and the @ line is shorthand for that replacement.
@decorator
def f(): ...
# is exactly the same as
def f(): ...
f = decorator(f)Because the replacement is usually a closure (section 1) that calls the original, a decorator can add behavior before and after a call without changing the function's code.
A first decorator
import functools
import time
def timed(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
start = time.perf_counter()
try:
return func(*args, **kwargs)
finally:
print(f"{func.__name__} took {time.perf_counter() - start:.2f}s")
return wrapper
@timed
def slow_add(a, b):
"""Add two numbers."""
time.sleep(0.01)
return a + b
print(slow_add(1, 2)) # slow_add took 0.01s, then 3The wrapper accepts *args, **kwargs so it works for any signature, returns whatever the function returns, and uses try/finally so the timing prints even if the function raises.
Always use functools.wraps
Without it, the decorated function silently becomes the wrapper, which breaks help text, debugging, and tools that read names or docstrings.
def no_wraps(func):
def wrapper(*a, **k):
return func(*a, **k)
return wrapper
@no_wraps
def hello():
"""Say hi."""
print(hello.__name__, hello.__doc__) # wrapper None
print(slow_add.__name__, slow_add.__doc__) # slow_add Add two numbers.
print(slow_add.__wrapped__.__name__) # slow_add: wraps keeps a link to the original@functools.wraps(func) copies __name__, __doc__, __module__, and related attributes onto the wrapper, updates its __dict__, and sets __wrapped__.
Decorators with arguments
When a decorator takes settings, as in @repeat(3), you need one more layer: a function that takes the settings and returns the real decorator. repeat(3) is evaluated first, and the result is then applied to the function.
def repeat(times): # layer 1: takes the settings
def decorator(func): # layer 2: takes the function
@functools.wraps(func)
def wrapper(*a, **k): # layer 3: runs on every call
result = None
for _ in range(times):
result = func(*a, **k)
return result
return wrapper
return decorator
@repeat(3)
def ping():
print("ping")
ping() # ping, ping, ping@repeat(3) means ping = repeat(3)(ping). A frequent mistake is writing @repeat without parentheses; then repeat receives the function as times, and the failure appears later.
Supporting both @deco and @deco(...)
A small pattern lets one decorator work bare and with arguments, which is how the lab's timed works:
def logged(func=None, *, level="INFO"):
def decorate(f):
@functools.wraps(f)
def wrapper(*a, **k):
print(f"[{level}] calling {f.__name__}")
return f(*a, **k)
return wrapper
return decorate if func is None else decorate(func)
@logged
def first(): return 1
@logged(level="DEBUG")
def second(): return 2
first(); second()
# [INFO] calling first
# [DEBUG] calling secondThe bare form passes the function as func; the call form leaves func empty, so the decorator returns decorate for Python to apply. The * makes level keyword-only, which prevents a function from being mistaken for a setting.
Class-based decorators
Any callable can be a decorator. A class gives you somewhere natural to keep state: the instance replaces the function, and __call__ runs on every call.
class CountCalls:
def __init__(self, func):
functools.update_wrapper(self, func) # like wraps, for an object
self.func = func
self.calls = 0
def __call__(self, *args, **kwargs):
self.calls += 1
return self.func(*args, **kwargs)
@CountCalls
def say(msg):
return msg
say("a"); say("b")
print(say.calls, say.__name__) # 2 sayA class-based decorator has one trap: on a method it fails, because an instance of your class is not a function and so is not a descriptor (section 4); Python never binds self.
class Bad:
@CountCalls
def m(self): return "m"
Bad().m() # TypeError: Bad.m() missing 1 required positional argument: 'self'The fix is to give the decorator class a __get__ that returns a bound method:
import types
class CountCalls2(CountCalls):
def __get__(self, instance, owner=None):
if instance is None:
return self # accessed on the class
return types.MethodType(self, instance)
class Good:
@CountCalls2
def m(self): return "m"
g = Good()
print(g.m(), g.m(), Good.m.calls) # m m 2Prefer a function decorator when you do not need state on the decorator object itself; it works on methods with no extra code.
Stacking decorators
With more than one decorator, the one closest to the def wraps first, so @a over @b means f = a(b(f)). At call time the outermost wrapper runs first.
def tag(name):
def deco(f):
print(f"decorating with {name}")
@functools.wraps(f)
def wrapper(*a, **k):
print(f"enter {name}")
r = f(*a, **k)
print(f"exit {name}")
return r
return wrapper
return deco
@tag("outer")
@tag("inner")
def work():
print("work")
work()The left side of the drawing shows the wrapping, and the right side shows exactly what the code above prints, in order.
Order matters in practice. In this week's lab, ttl_cache sits closest to the function so that exceptions are never cached, retry wraps it, and timed is outermost so it measures every attempt together.
The functools toolbox
Tool | What it does |
|---|---|
| Copy metadata from the wrapped function onto a wrapper |
| Freeze some arguments of a function, producing a new callable |
| Memoize results by arguments ( |
| Compute an attribute once per instance, then store it |
| Choose an implementation by the type of the first argument |
| Fold a sequence into one value with a two-argument function |
| Fill in comparison methods from |
import functools
from functools import partial
@functools.lru_cache(maxsize=None)
def fib(n):
return n if n < 2 else fib(n - 1) + fib(n - 2)
print(fib(30)) # 832040, computed with only 31 distinct calls
print(fib.cache_info()) # CacheInfo(hits=28, misses=31, maxsize=None, currsize=31)
fib.cache_clear() # empties the cache and resets the counters
int2 = partial(int, base=2)
print(int2("101")) # 5
@functools.singledispatch
def describe(x):
return f"object {x!r}"
@describe.register
def _(x: int): return f"int {x}"
@describe.register
def _(x: list): return f"list of {len(x)}"
print(describe(3), describe([1, 2]), describe("s")) # int 3 list of 2 object 's'
class Circle:
def __init__(self, r): self.r = r
@functools.cached_property
def area(self):
print("computing")
return 3.14159 * self.r ** 2
c = Circle(2)
print(c.area, c.area) # computing, then 12.56636 12.56636
print(functools.reduce(lambda acc, x: acc * x, [1, 2, 3, 4], 1)) # 24Two cautions for lru_cache: all arguments must be hashable, and on a method it keeps every self alive because self is part of the cache key. cached_property is a non-data descriptor that stores its result in the instance __dict__, as the descriptors section built by hand.
3. Context managers
A context manager guarantees that setup and cleanup code runs around a block, even if the block raises an exception.
The with statement is the tool: with open(path) as f: closes the file no matter how the block ends. Any object with __enter__ and __exit__ methods can be used this way.
Writing one as a class
import time
class Timer:
def __enter__(self):
self.start = time.perf_counter()
return self # becomes the name after `as`
def __exit__(self, exc_type, exc, tb):
self.elapsed = time.perf_counter() - self.start
return False # False: do not swallow exceptions
with Timer() as t:
sum(range(10_000))
print(t.elapsed)__exit__ receives the exception type, value, and traceback, or three None values when the block finished normally. Its return value matters:
- Returning a falsy value (the default
NoneorFalse) lets any exception continue propagating - Returning
Truesuppresses the exception, and execution continues after thewithblock
class Swallow:
def __enter__(self): return self
def __exit__(self, exc_type, exc, tb):
print("exit saw:", exc_type.__name__ if exc_type else None)
return exc_type is ZeroDivisionError # swallow only this one
with Swallow():
1 / 0 # exit saw: ZeroDivisionError
print("continued") # reached: the error was suppressed
with Swallow():
raise KeyError("k") # exit saw: KeyError, then KeyError propagatesSuppress exceptions sparingly and only specific types; swallowing everything hides bugs.
Writing one with contextlib.contextmanager
For most cases a generator is shorter than a class. Code before yield is the setup, the yielded value goes to as, and code after yield is the cleanup.
from contextlib import contextmanager
@contextmanager
def tag(name):
print(f"<{name}>")
try:
yield
finally:
print(f"</{name}>")
with tag("p"):
print("hello")
# <p>
# hello
# </p>An exception in the block is raised at the yield line. Always wrap the yield in try/finally when cleanup must run on errors; without it, the cleanup is skipped:
@contextmanager
def tag_no_finally(name):
print(f"<{name}>")
yield # an exception raised in the block surfaces here
print(f"</{name}>") # never reached when the block raisesA practical example that restores state:
import os
@contextmanager
def temp_chdir(path):
old = os.getcwd()
os.chdir(path)
try:
yield path
finally:
os.chdir(old)Useful tools in contextlib
Tool | What it does |
|---|---|
| Ignores the listed exceptions inside the block |
| Sends |
| Calls |
| Does nothing; a stand-in when a context manager is optional |
| Manages a dynamic number of context managers and cleanup callbacks |
| Base class that lets a context manager also work as a decorator |
| The |
import io
from contextlib import suppress, redirect_stdout, nullcontext
with suppress(FileNotFoundError):
os.remove("/nonexistent/file") # no error
buf = io.StringIO()
with redirect_stdout(buf):
print("captured")
print(repr(buf.getvalue())) # 'captured\n'
def maybe_lock(lock=None):
return lock if lock is not None else nullcontext()
with maybe_lock(): # works with or without a real lock
print("no lock needed")ExitStack: a variable number of resources
A with statement names its resources in the source code. When the number is only known at runtime, use ExitStack; it closes everything in reverse order, even if opening one of them fails.
from contextlib import ExitStack
paths = ["a.txt", "b.txt", "c.txt"]
with ExitStack() as stack:
files = [stack.enter_context(open(p, "w")) for p in paths]
stack.callback(print, "callback ran") # arbitrary cleanup, runs last-in first-out
for i, f in enumerate(files):
f.write(str(i))
print([f.closed for f in files]) # [True, True, True]One object, two roles: ContextDecorator
Subclass ContextDecorator and the same class works as a with block and as a function decorator. The library in this week's lab uses this for its Timer.
from contextlib import ContextDecorator
class Tracer(ContextDecorator):
def __enter__(self):
print("start"); return self
def __exit__(self, *exc):
print("stop"); return False
@Tracer()
def job():
print("working")
job() # start, working, stopA function decorated with @contextmanager also gets this ability: tag("p") can be used both ways.
4. Descriptors
A descriptor is an object stored as a class attribute that controls what happens when an instance attribute is read, set, or deleted.
Descriptors are the machinery behind property, methods, classmethod, staticmethod, and __slots__. Learning them explains how Python attribute access really works.
A descriptor is any object whose class defines one or more of these methods:
Method | Called when | Signature |
|---|---|---|
| The attribute is read |
|
| The attribute is assigned on an instance |
|
| The attribute is deleted with |
|
| The owning class is created |
|
When __get__ is called on the class itself (Item.price), obj is None; convention is to return the descriptor in that case.
A reusable validated attribute
The descriptor lives on the class and is shared by all instances, so it must keep each instance's data on the instance. __set_name__ tells it the attribute name it was assigned to.
class Positive:
def __set_name__(self, owner, name):
self.private = "_" + name # e.g. "_price"
def __get__(self, obj, objtype=None):
if obj is None:
return self # accessed on the class
return getattr(obj, self.private)
def __set__(self, obj, value):
if value <= 0:
raise ValueError(f"{self.private[1:]} must be positive, got {value}")
setattr(obj, self.private, value)
class Item:
price = Positive()
qty = Positive()
def __init__(self, price, qty):
self.price = price # goes through Positive.__set__
self.qty = qty
it = Item(9.5, 2)
print(it.price, it.qty) # 9.5 2
print(vars(it)) # {'_price': 9.5, '_qty': 2}
it.qty = 0 # ValueError: qty must be positive, got 0One Positive class now validates any number of attributes in any number of classes, which is the advantage over writing a property pair for each one.
Do not store per-instance values on the descriptor itself (for example self.value = value): one descriptor object serves every instance, so all instances would share and overwrite that single value.
Data and non-data descriptors
Descriptors that define __set__ or __delete__ are data descriptors; those that define only __get__ are non-data descriptors. The difference decides which wins when the instance also has a value with the same name.
For obj.name, Python checks in this order:
- A data descriptor on the class (or its parents) wins
- Otherwise, an entry in the instance's
__dict__ - Otherwise, a non-data descriptor or plain attribute on the class
- Otherwise,
__getattr__if the class defines it, orAttributeError
class DataD:
def __get__(self, o, t=None): return "descriptor"
def __set__(self, o, v): pass
class NonData:
def __get__(self, o, t=None): return "descriptor"
class Demo:
d = DataD()
n = NonData()
dm = Demo()
dm.__dict__["d"] = "instance"
dm.__dict__["n"] = "instance"
print(dm.d, dm.n) # descriptor instanceFunctions are descriptors: how methods work
Every function defined in a class body is a non-data descriptor. Its __get__ returns a bound method that remembers the instance, which is why self is passed automatically.
class A:
def hello(self): return "hi"
a = A()
print(type(A.__dict__["hello"]).__name__) # function
bound = A.hello.__get__(a, A) # what a.hello does behind the scenes
print(bound(), bound.__self__ is a) # hi True
a.hello = lambda: "shadow" # allowed: non-data descriptors can be shadowed
print(a.hello()) # shadowThis is also why a class-based decorator needs a __get__ method to work on methods, as the decorators section showed.
property is a data descriptor
class Temp:
def __init__(self):
self._c = 0
@property
def celsius(self):
return self._c
@celsius.setter
def celsius(self, value):
if value < -273.15:
raise ValueError("below absolute zero")
self._c = value
t = Temp(); t.celsius = 25
print(t.celsius) # 25
print(type(Temp.__dict__["celsius"]).__name__) # propertyUse property for one attribute in one class, and write your own descriptor when the same logic applies to many attributes.
A lazy attribute: a non-data descriptor that caches
Because a non-data descriptor loses to the instance __dict__, it can compute a value once, store it there, and never run again. This is how functools.cached_property works.
class Lazy:
def __init__(self, func):
self.func = func
self.name = func.__name__
def __get__(self, obj, objtype=None):
if obj is None:
return self
value = self.func(obj)
obj.__dict__[self.name] = value # shadows the descriptor from now on
return value
class Report:
@Lazy
def data(self):
print("loading...")
return [1, 2, 3]
r = Report()
print(r.data) # loading... then [1, 2, 3]
print(r.data) # [1, 2, 3]: no second load
print("data" in vars(r)) # True5. Metaclasses and __init_subclass__
A metaclass is the class of a class: it controls how a class object is created, the same way a class controls how its instances are created.
Classes are objects, so something must create them. By default that something is type.
print(type(42), type(int), type(type)) # <class 'int'> <class 'type'> <class 'type'>type can also build a class directly from a name, a tuple of base classes, and a dictionary of attributes. A class statement is a convenient way to call it:
Point = type("Point", (), {"x": 0, "hello": lambda self: "hi"})
p = Point()
print(Point.__name__, p.x, p.hello()) # Point 0 hiWhat happens when a class statement runs
Python runs these steps in order. The numbers are the order printed by the experiment below.
- The metaclass's
__prepare__returns the namespace dict for the class body - The class body executes, filling that namespace
- The metaclass is called with the name, bases, and namespace:
Meta.__new__creates the class object - Inside
type.__new__, Python calls__set_name__on every descriptor in the namespace, then__init_subclass__on the parent class Meta.__init__initializes the new class- Later, calling the class to make an instance runs
Meta.__call__, which in turn runs__new__and__init__of the class
class Meta(type):
@classmethod
def __prepare__(mcls, name, bases, **kw):
print("1 __prepare__", name)
return super().__prepare__(name, bases, **kw)
def __new__(mcls, name, bases, ns, **kw):
print("3 Meta.__new__", name)
return super().__new__(mcls, name, bases, ns)
def __init__(cls, name, bases, ns, **kw):
print("5 Meta.__init__", name)
super().__init__(name, bases, ns)
def __call__(cls, *a, **k):
print("6 Meta.__call__ (making an instance)")
return super().__call__(*a, **k)
class Desc:
def __set_name__(self, owner, name):
print("4a __set_name__", name)
class Base:
def __init_subclass__(cls, **kw):
print("4b __init_subclass__", cls.__name__)
super().__init_subclass__(**kw)
class Thing(Base, metaclass=Meta):
print("2 class body runs")
d = Desc()
obj = Thing()Output, in order:
1 __prepare__ Thing
2 class body runs
3 Meta.__new__ Thing
4a __set_name__ d
4b __init_subclass__ Thing
5 Meta.__init__ Thing
6 Meta.__call__ (making an instance)Example: a singleton metaclass
Overriding the metaclass's __call__ controls instance creation.
class Singleton(type):
_instances = {}
def __call__(cls, *a, **k):
if cls not in cls._instances:
cls._instances[cls] = super().__call__(*a, **k)
return cls._instances[cls]
class Config(metaclass=Singleton):
def __init__(self):
print("init runs once")
print(Config() is Config()) # init runs once, then TruePrefer __init_subclass__ for most jobs
Metaclasses are powerful but heavy: a class can have only one metaclass, and combining libraries that each bring their own causes conflicts. Since Python 3.6, __init_subclass__ covers the most common reason to want one, which is reacting when someone subclasses your class.
class Plugin:
registry = {}
def __init_subclass__(cls, /, key=None, **kwargs):
super().__init_subclass__(**kwargs)
cls.registry[key or cls.__name__.lower()] = cls
class CsvPlugin(Plugin): ...
class JsonPlugin(Plugin, key="json"): ...
print(Plugin.registry)
# {'csvplugin': <class '__main__.CsvPlugin'>, 'json': <class '__main__.JsonPlugin'>}The key keyword in the class line is passed straight to __init_subclass__. The same hook can enforce rules on subclasses:
class Task:
required = ("run",)
def __init_subclass__(cls, **kw):
super().__init_subclass__(**kw)
missing = [n for n in cls.required if not callable(getattr(cls, n, None))]
if missing:
raise TypeError(f"{cls.__name__} must define {missing}")
class Good(Task):
def run(self): ...
class Bad(Task): ... # TypeError: Bad must define ['run']Choosing the right tool
You want to | Use |
|---|---|
React to subclass creation: register it, validate it, set defaults |
|
Modify or register one class | A class decorator (a function that takes and returns a class) |
React when a descriptor is attached to a class |
|
Control instance creation (singleton, pooling) | A metaclass |
Change the class namespace before the body runs, or alter the class creation itself | A metaclass ( |
Metaclasses you already use: abc.ABCMeta (abstract classes), enum.EnumMeta (enums), and the metaclasses behind many ORMs and web frameworks. Writing your own is rare; recognizing them when you read library code is the useful skill.
6. Dataclasses
The @dataclass decorator writes __init__, __repr__, and __eq__ for you from a list of annotated fields, so classes that mainly hold data need no boilerplate.
This decorator uses the dunder methods and this week's decorator idea: it takes a class, adds methods to it, and returns it.
from dataclasses import dataclass, asdict
@dataclass
class Point:
x: float
y: float = 0.0 # a field with a default
p = Point(1.5)
print(p) # Point(x=1.5, y=0.0)
print(p == Point(1.5, 0.0)) # True: __eq__ compares the fields as a tuple
print(asdict(p)) # {'x': 1.5, 'y': 0.0}Fields without defaults must come before fields with defaults, as with function parameters.
Mutable defaults need default_factory
The mutable default trap applies here, and dataclass refuses it outright:
from dataclasses import field
@dataclass
class BadD:
items: list = []
# ValueError: mutable default <class 'list'> for field items is not allowed: use default_factory
@dataclass
class Basket:
items: list = field(default_factory=list) # a fresh list per instance
a, b = Basket(), Basket()
a.items.append(1)
print(a, b) # Basket(items=[1]) Basket(items=[])Options on the decorator
Option | Effect |
|---|---|
| Adds |
| Makes instances immutable and, with |
| Uses |
| Makes every field keyword-only in |
| Keeps identity-based equality instead of generating |
@dataclass(order=True, frozen=True, slots=True)
class Version:
major: int
minor: int = 0
tags: tuple = field(default=(), compare=False) # left out of ==, <, and hash
v1, v2 = Version(1, 2), Version(1, 10)
print(v1 < v2) # True: compares (1, 2) with (1, 10)
print(sorted([v2, v1])) # [Version(major=1, minor=2, ...), Version(major=1, minor=10, ...)]
print(hash(v1) == hash(Version(1, 2))) # True: frozen + eq makes it hashable
print(hasattr(v1, "__dict__")) # False: slots
v1.major = 5 # FrozenInstanceError: cannot assign to field 'major'A regular (non-frozen) dataclass with eq=True sets __hash__ to None, so instances are unhashable; this is the __eq__-without-__hash__ rule:
hash(Point(1)) # TypeError: unhashable type: 'Point'To "change" a frozen instance, create a modified copy with replace:
from dataclasses import replace
print(replace(v1, minor=9)) # Version(major=1, minor=9, tags=())Computed fields: __post_init__, InitVar, init=False
__post_init__ runs right after the generated __init__, so use it for validation and derived values. An InitVar is passed to __init__ and __post_init__ but is not stored as a field.
from dataclasses import InitVar
@dataclass
class Rect:
w: float
h: float
scale: InitVar[float] = 1 # input only, not stored
area: float = field(init=False) # computed, not a constructor argument
def __post_init__(self, scale):
self.w *= scale
self.h *= scale
self.area = self.w * self.h
print(Rect(2, 3, scale=2)) # Rect(w=4, h=6, area=24)Validation belongs in __post_init__ too:
@dataclass
class Emp:
name: str
def __post_init__(self):
if not self.name:
raise ValueError("name required")Keyword-only fields and inheritance
@dataclass(kw_only=True)
class Cfg:
host: str = "localhost"
port: int = 8080
print(Cfg(port=9000)) # Cfg(host='localhost', port=9000)
Cfg("x") # TypeError: Cfg.__init__() takes 1 positional argument but 2 were given
@dataclass
class Base2:
a: int
@dataclass
class Child(Base2): # fields from the base come first
b: int = 0
print(Child(1, 2)) # Child(a=1, b=2)Inspect a dataclass with fields(cls) and asdict(obj); both are in the dataclasses module.
When to use a dataclass
- Dataclass: a typed record you create in your own code, where you want defaults, ordering, or immutability
NamedTuple: an immutable record that must also behave as a tuple (unpacking, indexing)- Plain class: the object has behavior and invariants that matter more than its fields
pydanticmodel: data arrives from outside (JSON, user input) and must be validated and converted; dataclasses do not check types at runtime
7. Protocol and ABCs
Both tools describe an interface; an ABC enforces it by inheritance at runtime, while a Protocol describes it by shape and is checked by type checkers.
Abstract base classes (nominal typing)
An ABC lists methods that subclasses must implement. A class that inherits from the ABC but leaves an @abstractmethod undefined cannot be instantiated.
from abc import ABC, abstractmethod
import math
class Shape(ABC):
@abstractmethod
def area(self) -> float: ...
def describe(self) -> str: # concrete method shared by all shapes
return f"{type(self).__name__} with area {self.area():.2f}"
class Circle(Shape):
def __init__(self, r): self.r = r
def area(self): return math.pi * self.r ** 2
class Broken(Shape): pass
Shape() # TypeError: Can't instantiate abstract class Shape without an implementation for abstract method 'area'
Broken() # TypeError: Can't instantiate abstract class Broken without an implementation for abstract method 'area'
print(Circle(1).describe()) # Circle with area 3.14The error happens at instantiation, so a missing method is caught the first time anyone builds the object, not when the missing method is called deep inside a program.
Useful ABC features:
- Abstract properties: stack
@propertyon top of@abstractmethod - Mixin methods: concrete methods built on the abstract ones, like
describeabove (this is howcollections.abc.Sequencegave you__iter__and__contains__. - Virtual subclasses:
Shape.register(Tri)makesisinstance(Tri(), Shape)true without inheritance
class Tri:
def area(self): return 0.5
print(isinstance(Tri(), Shape)) # False
Shape.register(Tri)
print(isinstance(Tri(), Shape)) # True: registered, but Tri inherits nothingProtocols (structural typing)
A Protocol says "anything with these methods qualifies", with no inheritance required. This is static duck typing: the class Resource below never mentions SupportsClose.
from typing import Protocol
class SupportsClose(Protocol):
def close(self) -> None: ...
class Resource:
def close(self) -> None:
print("closed")
class Wrong:
def shut(self) -> None: ...
def shutdown(item: SupportsClose) -> None:
item.close()
shutdown(Resource()) # accepted: has close()
shutdown(Wrong()) # rejected by the type checkerRunning mypy on that file reports:
error: Argument 1 to "shutdown" has incompatible type "Wrong"; expected "SupportsClose" [arg-type]Python itself does not stop shutdown(Wrong()); it fails at runtime with an AttributeError on item.close(). A protocol's value comes from the checker, run in your editor and CI.
Add @runtime_checkable if you also want isinstance to test for the methods at runtime:
from typing import runtime_checkable
import io
@runtime_checkable
class SupportsClose(Protocol):
def close(self) -> None: ...
print(isinstance(io.StringIO(), SupportsClose)) # True
print(isinstance(42, SupportsClose)) # False
print(isinstance(Wrong(), SupportsClose)) # FalseA runtime check only confirms that the method names exist; it does not check argument types or return types.
Choosing between them
Question | ABC | Protocol |
|---|---|---|
Must implementers inherit from it? | Yes (or be registered) | No |
Can it supply shared, concrete methods? | Yes | Not usefully; it is an interface only |
Where is a violation caught? | At instantiation, at runtime | By the type checker, before running |
Works with classes you do not control? | Only through | Yes, automatically |
Typical use | A framework base class you want people to extend | A parameter type that accepts anything shaped right |
A good rule: use a Protocol to type the inputs of your functions, so callers can pass anything compatible, and use an ABC for a base class that carries shared behavior and needs enforced overrides.
Lab: Build a retry, caching, and timing decorator library
You will build decorkit.py, a small library of five tools that use every idea from this week, and prove it correct with 13 pytest tests that run in milliseconds. The code below passed all 13 tests on Python 3.13.
Specification
Tool | Kind | Behavior |
|---|---|---|
| Class using | Works as a |
| Function decorator, bare or with arguments | Records |
| Decorator with arguments | Retries on chosen exceptions with |
| Decorator with arguments | Caches results for |
| Class-based decorator | Counts calls; works on plain functions and on methods |
Design decisions to notice as you read the code:
- Time is injected.
timed,Timer, andttl_cachetake aclock, andretrytakes asleep, so tests control time instead of waiting for it - Every wrapper uses
functools.wraps, so names, docstrings, and__wrapped__survive - State lives on the wrapper function (
calls,cache_info). Becausewrapscopies a function's__dict__, these attributes stay visible when decorators are stacked - Failures are handled deliberately:
timedrecords time in afinally,retryre-raises the last error, andttl_cachenever stores an exception
Step 1: Timer and timed
Create decorkit.py and start with the imports, Timer, and timed.
"""Week 2 lab: a small decorator library with timed, retry, ttl_cache and CallCounter."""
from __future__ import annotations
import functools
import threading
import time
import types
from collections import OrderedDict
from contextlib import ContextDecorator
from typing import Any, Callable, NamedTuple
class Timer(ContextDecorator):
"""Measure elapsed time; works as a `with` block and as a decorator."""
def __init__(self, clock: Callable[[], float] = time.perf_counter):
self._clock = clock
self.elapsed = 0.0
def __enter__(self) -> Timer:
self._start = self._clock()
return self
def __exit__(self, exc_type, exc, tb) -> bool:
self.elapsed = self._clock() - self._start
return False
def timed(func=None, *, report: Callable[[str, float], None] | None = None,
clock: Callable[[], float] = time.perf_counter):
"""Record call count and timing on the wrapper; optionally report every call.
Works bare (`@timed`) and with arguments (`@timed(report=print)`).
"""
def decorate(fn):
@functools.wraps(fn)
def wrapper(*args, **kwargs):
start = clock()
try:
return fn(*args, **kwargs)
finally:
elapsed = clock() - start
wrapper.calls += 1
wrapper.last_elapsed = elapsed
wrapper.total_elapsed += elapsed
if report is not None:
report(fn.__name__, elapsed)
wrapper.calls = 0
wrapper.last_elapsed = 0.0
wrapper.total_elapsed = 0.0
return wrapper
return decorate if func is None else decorate(func)Try it (timings will differ slightly on your machine):
>>> from decorkit import timed, Timer
>>> import time
>>> @timed(report=lambda name, s: print(f"{name}: {s:.3f}s"))
... def nap():
... time.sleep(0.05)
>>> nap()
nap: 0.050s
>>> nap.calls, round(nap.total_elapsed, 2)
(1, 0.05)
>>> with Timer() as t:
... sum(range(1_000_000))
>>> t.elapsed > 0
TrueThe pattern return decorate if func is None else decorate(func) is the optional-arguments trick from the decorators section: with @timed(...) the first argument is empty, so a decorator is returned; with bare @timed, the function arrives directly.
Step 2: retry
This is a decorator factory: three nested functions, where the outermost takes the settings, the middle takes the function, and the innermost is the wrapper that runs on every call. Append it to decorkit.py.
def retry(attempts: int = 3, *, delay: float = 0.0, backoff: float = 1.0,
exceptions: type[BaseException] | tuple[type[BaseException], ...] = Exception,
on_retry: Callable[[int, BaseException], None] | None = None,
sleep: Callable[[float], None] = time.sleep):
"""Retry a function that raises one of `exceptions`, up to `attempts` calls in total.
The wait starts at `delay` and is multiplied by `backoff` after each failure.
The last exception is re-raised once the attempts are used up.
"""
if attempts < 1:
raise ValueError("attempts must be at least 1")
def decorate(fn):
@functools.wraps(fn)
def wrapper(*args, **kwargs):
wait = delay
for attempt in range(1, attempts + 1):
try:
return fn(*args, **kwargs)
except exceptions as exc:
if attempt == attempts:
raise
if on_retry is not None:
on_retry(attempt, exc)
if wait > 0:
sleep(wait)
wait *= backoff
return wrapper
return decorateWhat to notice:
- The
attempts < 1check runs when the decorator is applied, not when the function is called, so a bad setting fails at import time - A bare
raiseinsideexceptre-raises the original exception with its traceback intact - With
delay=1, backoff=2the waits are 1 s, then 2 s, then 4 s, and so on - Only exceptions listed in
exceptionsare retried; anything else escapes on the first call
>>> from decorkit import retry
>>> import random
>>> @retry(attempts=4, delay=0.1, backoff=2, exceptions=ConnectionError,
... on_retry=lambda n, e: print(f"attempt {n} failed: {e}"))
... def call_api():
... if random.random() < 0.7:
... raise ConnectionError("timeout")
... return "ok"Step 3: ttl_cache
This combines a closure (the cache lives in the enclosing scope), a lock, and an OrderedDict that doubles as a least-recently-used list. Append it to decorkit.py.
class CacheInfo(NamedTuple):
hits: int
misses: int
maxsize: int
currsize: int
def ttl_cache(ttl: float = 60.0, maxsize: int = 128,
clock: Callable[[], float] = time.monotonic):
"""Cache results for `ttl` seconds, keeping at most `maxsize` entries (least recently used out).
Arguments must be hashable. Exceptions are not cached. Thread-safe.
"""
if ttl <= 0:
raise ValueError("ttl must be positive")
if maxsize < 1:
raise ValueError("maxsize must be at least 1")
def decorate(fn):
store: OrderedDict[Any, tuple[float, Any]] = OrderedDict()
lock = threading.Lock()
stats = {"hits": 0, "misses": 0}
@functools.wraps(fn)
def wrapper(*args, **kwargs):
key = (args, tuple(sorted(kwargs.items())))
with lock:
entry = store.get(key)
if entry is not None and clock() - entry[0] < ttl:
store.move_to_end(key)
stats["hits"] += 1
return entry[1]
stats["misses"] += 1
value = fn(*args, **kwargs) # computed outside the lock so slow calls don't block others
with lock:
store[key] = (clock(), value)
store.move_to_end(key)
while len(store) > maxsize:
store.popitem(last=False)
return value
def cache_info() -> CacheInfo:
with lock:
return CacheInfo(stats["hits"], stats["misses"], maxsize, len(store))
def cache_clear() -> None:
with lock:
store.clear()
stats["hits"] = stats["misses"] = 0
wrapper.cache_info = cache_info
wrapper.cache_clear = cache_clear
return wrapper
return decorateHow the pieces fit:
store,lock, andstatsare free variables captured bywrapper,cache_info, andcache_clear: one private cache per decorated function (closures)- The key is built from positional arguments plus sorted keyword items, so
add(1, b=2)andadd(1, b=3)are different entries move_to_endmarks an entry as most recently used, andpopitem(last=False)removes the oldest- The lock is released while
fnruns, so a slow call does not block other threads; the trade-off is that two threads can compute the same missing value at once cache_infois aNamedTuple, so it compares equal to a plain tuple, which the tests use
>>> from decorkit import ttl_cache
>>> @ttl_cache(ttl=5, maxsize=2)
... def square(x):
... print("computing", x)
... return x * x
>>> square(4); square(4)
computing 4
16
16
>>> square.cache_info()
CacheInfo(hits=1, misses=1, maxsize=2, currsize=1)Step 4: CallCounter (class-based)
A class used as a decorator stores the function and its state in an instance and implements __call__. On its own it breaks on methods, because an instance is not a function and so is not a descriptor; adding __get__ fixes that, as covered in the descriptors section.
class CallCounter:
"""Class-based decorator that counts calls; also works on methods."""
def __init__(self, func):
functools.update_wrapper(self, func)
self._func = func
self.calls = 0
def __call__(self, *args, **kwargs):
self.calls += 1
return self._func(*args, **kwargs)
def __get__(self, instance, owner=None):
if instance is None:
return self
return types.MethodType(self, instance)functools.update_wrapper is what wraps calls internally; use it directly when the wrapper is an object rather than a function. types.MethodType(self, instance) builds a bound method that passes instance as the first argument. The counter is per decorated function, so calls from every instance of the class add up.
Step 5: Prove it with tests
Save this as test_decorkit.py next to decorkit.py, install the runner with pip install pytest, and run pytest -q. All 13 tests should pass.
The FakeClock is the key technique: it is a callable that returns a number you move forward by hand, so expiry, timing, and backoff are tested without sleeping.
import pytest
from decorkit import CallCounter, Timer, retry, timed, ttl_cache
class FakeClock:
"""A controllable clock so tests never sleep."""
def __init__(self):
self.now = 0.0
def __call__(self):
return self.now
def advance(self, seconds):
self.now += seconds
# --- Timer and timed ----------------------------------------------------------
def test_timer_as_context_manager_and_decorator():
clock = FakeClock()
with Timer(clock) as t:
clock.advance(1.5)
assert t.elapsed == 1.5
timer = Timer(clock)
@timer
def work():
clock.advance(2)
work()
assert timer.elapsed == 2
def test_timed_records_calls_and_elapsed():
clock = FakeClock()
@timed(clock=clock)
def work():
"""Do work."""
clock.advance(2.5)
return "done"
assert work() == "done"
assert work.calls == 1 and work.last_elapsed == 2.5
work()
assert work.calls == 2 and work.total_elapsed == 5.0
assert work.__name__ == "work" and work.__doc__ == "Do work."
def test_timed_works_bare_and_reports_even_on_failure():
@timed
def one():
return 1
assert one() == 1 and one.calls == 1
reports = []
@timed(report=lambda name, elapsed: reports.append(name), clock=FakeClock())
def boom():
raise RuntimeError("x")
with pytest.raises(RuntimeError):
boom()
assert reports == ["boom"] and boom.calls == 1
# --- retry ----------------------------------------------------------------------
def test_retry_succeeds_after_failures_with_backoff():
calls, sleeps = [], []
@retry(attempts=3, delay=1, backoff=2, sleep=sleeps.append)
def flaky():
calls.append(1)
if len(calls) < 3:
raise ConnectionError("down")
return "ok"
assert flaky() == "ok"
assert len(calls) == 3
assert sleeps == [1, 2]
def test_retry_reraises_last_error_and_calls_hook():
seen = []
@retry(attempts=2, on_retry=lambda n, e: seen.append((n, str(e))), sleep=lambda s: None)
def always_fails():
raise ValueError("bad")
with pytest.raises(ValueError, match="bad"):
always_fails()
assert seen == [(1, "bad")]
def test_retry_only_catches_listed_exceptions():
calls = []
@retry(attempts=3, exceptions=(KeyError,), sleep=lambda s: None)
def wrong_kind():
calls.append(1)
raise TypeError("not retried")
with pytest.raises(TypeError):
wrong_kind()
assert len(calls) == 1
def test_retry_validates_attempts():
with pytest.raises(ValueError):
retry(attempts=0)
# --- ttl_cache ------------------------------------------------------------------
def test_ttl_cache_hits_and_expiry():
clock = FakeClock()
calls = []
@ttl_cache(ttl=10, clock=clock)
def square(x):
calls.append(x)
return x * x
assert square(3) == 9 and square(3) == 9
assert calls == [3]
assert square.cache_info() == (1, 1, 128, 1)
clock.advance(11)
assert square(3) == 9
assert calls == [3, 3]
def test_ttl_cache_evicts_least_recently_used():
calls = []
@ttl_cache(ttl=100, maxsize=2, clock=FakeClock())
def ident(x):
calls.append(x)
return x
ident(1); ident(2); ident(1); ident(3) # 2 is the least recently used: evicted
ident(1) # still cached
ident(2) # was evicted: computed again
assert calls == [1, 2, 3, 2]
def test_ttl_cache_keys_include_keyword_arguments_and_clear():
calls = []
@ttl_cache(ttl=100, clock=FakeClock())
def add(a, b=0):
calls.append((a, b))
return a + b
add(1, b=2); add(1, b=2); add(1, b=3)
assert calls == [(1, 2), (1, 3)]
add.cache_clear()
assert add.cache_info().currsize == 0
def test_ttl_cache_does_not_cache_exceptions():
calls = []
@ttl_cache(ttl=100, clock=FakeClock())
def fragile():
calls.append(1)
raise OSError("nope")
for _ in range(2):
with pytest.raises(OSError):
fragile()
assert len(calls) == 2
# --- CallCounter ---------------------------------------------------------------
def test_call_counter_on_function_and_method():
@CallCounter
def add(a, b):
return a + b
assert add(1, 2) == 3 and add.calls == 1 and add.__name__ == "add"
class Greeter:
@CallCounter
def hi(self, name):
return f"hi {name}"
g = Greeter()
assert g.hi("bo") == "hi bo"
assert Greeter.hi.calls == 1
# --- composing decorators -------------------------------------------------------
def test_decorators_compose():
clock = FakeClock()
calls = []
@timed(clock=clock)
@retry(attempts=2, sleep=lambda s: None)
@ttl_cache(ttl=10, clock=clock)
def fetch(key):
calls.append(key)
if len(calls) == 1:
raise OSError("first call fails")
return key.upper()
assert fetch("a") == "A" # first attempt fails, the retry succeeds
assert calls == ["a", "a"]
assert fetch("a") == "A" # served from the cache
assert calls == ["a", "a"]
assert fetch.calls == 2 # counted by the outer timed wrapper
assert fetch.cache_info().hits == 1 # attributes survive functools.wrapsThe last test shows composition at work: ttl_cache sits closest to the function, so an exception is never cached and retry calls it again; timed is outermost, so it counts calls to the whole stack.
Stretch goals
- Make
timed,retry, andttl_cachework onasync deffunctions, usinginspect.iscoroutinefunctionto pick anasync defwrapper andasyncio.sleepfor the delay - Add jitter to
retry(a random extra wait of up to a given fraction of the delay) and test it by injecting a fake random function - Type the library so
mypy --strictpasses, usingParamSpecandTypeVarso decorated functions keep their signatures; the current untyped wrappers report errors under--strict - Add
@ttl_cachesupport for unhashable arguments by falling back toreprof the arguments, then write down the risks of that choice - Add a
circuit_breakerdecorator that stops calling a failing function for a cool-down period after N consecutive failures, tested withFakeClock - Publish the library with a
pyproject.tomland install it into a fresh virtual environment, as a preview of Week 6
Exercises, pitfalls, and checklist
Predict each output before running it; a wrong prediction points at the part of this tutorial to reread.
Predict the output
import functools
from contextlib import contextmanager
from dataclasses import dataclass
# 1
def make():
fs = []
for i in range(3):
fs.append(lambda i=i: i * 10)
return fs
print([f() for f in make()])
# 2
def a(f):
def w(): return "a(" + f() + ")"
return w
def b(f):
def w(): return "b(" + f() + ")"
return w
@a
@b
def f(): return "f"
print(f())
# 3
class D:
def __get__(self, o, t=None): return "desc"
class C:
x = D()
c = C(); c.x = "inst"
print(c.x)
# 4
@dataclass
class P:
x: int
print(P(1) == P(1), P(1) is P(1))
# 5
@contextmanager
def cm():
print("in")
yield
print("out")
with cm():
print("body")
# 6
def deco(func):
def wrapper(*a, **k):
return func(*a, **k)
return wrapper
@deco
def hello(): pass
print(hello.__name__)
# 7
@functools.lru_cache
def g(x):
print("run", x); return x
g(1); g(1); g(2)
print(g.cache_info().hits)# | Output | Why |
|---|---|---|
1 |
| The default argument |
2 |
| Decorators apply bottom-up ( |
3 |
|
|
4 |
| A dataclass generates |
5 |
| Code before |
6 |
| Without |
7 |
| The second |
Hands-on exercises
- Closure versus class: write
make_accumulator()that returns anadd(x)function and atotal()function sharing one running total throughnonlocal. Then rewrite it as a class and list one advantage of each version. @memoize: write your own cache decorator that supports keyword arguments and exposes the cache dict aswrapper.cache. Compare its behavior tofunctools.lru_cacheonfib(35).@deprecated("use new_func"): a decorator with an argument that issues aDeprecationWarning(withwarnings.warn,stacklevel=2) on each call and keeps the function's metadata. Test it withpytest.warns.temp_env(**variables): set environment variables inside awithblock and restore the originals afterward, including when the block raises. Write it twice, once as a class and once with@contextmanager.- Typed descriptors: build
Typed(kind)andRange(low, high)descriptors with__set_name__, and use them on aProductclass withname,price, andstockattributes. Check that bad values raiseTypeErrororValueError. - Plugin registry: write a base class whose subclasses register under a
name=class keyword through__init_subclass__, raising an error on duplicate names. Then redo it with a class decorator and compare. - Dataclass model: model an
Orderholding a list of frozenLineItemdataclasses, with a computedtotalfield set in__post_init__, and make orders sortable by total. - Protocol check: define
class Reader(Protocol): def read(self) -> str: ...and a functionload(source: Reader) -> str. Runmypyto confirm it accepts anio.StringIOand rejects anint.
Common pitfalls
Pitfall | Fix |
|---|---|
Forgetting | Add it to every wrapper; the name, docstring, and signature introspection depend on it |
Writing | Use the |
Late-binding closures in a loop | Capture with a default argument, |
Class-based decorator on a method fails with a missing | Implement |
| Wrap the |
Storing per-instance values on a descriptor object | Store them on the instance ( |
| The cache keeps every |
Mutable default in a dataclass field | Use |
Reaching for a metaclass first | Try |
Expecting | It only checks that the method names exist; use a type checker for the rest |
Caching calls with unhashable arguments | Convert them to tuples, or key the cache on an identifier |
Checklist
- I can explain what a free variable is, and when
nonlocalis required - I can write a decorator three ways: plain, with arguments, and class-based, each preserving metadata
- I can predict the order in which stacked decorators wrap and run
- I can write a context manager as a class and with
@contextmanager, and explain what returningTruefrom__exit__does - I can explain the lookup order for data descriptors, instance attributes, and non-data descriptors
- I can explain why
selfis passed automatically to a method - I can list the steps that run when a
classstatement executes - I can choose between a dataclass, a
NamedTuple, a plain class, and apydanticmodel - I can choose between an ABC and a
Protocolfor a given interface - My
decorkitlibrary passes all 13 tests, and I completed at least one stretch goal
Next week: iterators, generators, and functional patterns. Generators are built from the same ideas as this week's @contextmanager: a function that pauses at yield and resumes later.

No comments:
Post a Comment