Monday, November 9, 2026

Advanced Functions and Object-Oriented Python

Level: Intermediate to advanced

Suggested time: 5–7 hours
Python: 3.10+ (examples use standard-library features)

This week moves beyond writing functions and classes into shaping how they behave. You will learn how functions can retain state, how decorators wrap behavior, how Python manages resources and attribute access, and how classes can be customized at creation time. The lab brings these ideas together in a small reusable decorator library.

Learning goals

By the end, you should be able to:

  • Explain closures and use decorators with parameters.
  • Preserve a wrapped function's metadata with functools.
  • Implement function-based and class-based decorators.
  • Manage resources with context managers and contextlib.
  • Explain descriptors and use them to validate attributes.
  • Choose between dataclasses, protocols, abstract base classes, and class initialization hooks.
  • Recognize when a metaclass is justified and when a simpler hook is clearer.
  • Build and compose retry, caching, and timing decorators.

1. Closures: functions that remember

A closure is a function together with references to variables from its enclosing scope. The inner function can keep using those variables after the outer function returns.

def make_multiplier(factor):
    def multiply(value):
        return factor * value
    return multiply

triple = make_multiplier(3)
assert triple(7) == 21

multiply remembers factor. Python exposes captured variables through closure, though application code rarely needs to inspect it.

Closures are useful for small pieces of per-function state and are the foundation of many decorators. For mutable or rebinding state, use nonlocal:

def make_counter():
    count = 0

    def next_value():
        nonlocal count
        count += 1
        return count

    return next_value

counter = make_counter()
assert [counter(), counter(), counter()] == [1, 2, 3]

Without nonlocal, assigning to count inside next_value would make it a local variable and reading it first would fail with UnboundLocalError.

Late binding in loops

Closures capture variables, not a frozen copy of their current value. A common surprise:

functions = []
for number in range(3):
    functions.append(lambda: number)

assert [fn() for fn in functions] == [2, 2, 2]

All three functions look up the same number after the loop. Bind the current value with a default argument:

functions = [lambda number=number: number for number in range(3)]
assert [fn() for fn in functions] == [0, 1, 2]

2. Decorators and functools

A decorator accepts a callable and returns a callable, usually a wrapper that adds behavior before or after the original call.

from functools import wraps
from time import perf_counter

def timed(function):
    @wraps(function)
    def wrapper(*args, **kwargs):
        started = perf_counter()
        try:
            return function(*args, **kwargs)
        finally:
            elapsed = perf_counter() - started
            print(f"{function.__name__} took {elapsed:.6f}s")
    return wrapper

@timed
def add(a, b):
    """Add two values."""
    return a + b

@timed is equivalent to add = timed(add). @wraps copies useful metadata such as the function name, docstring, annotations, and wrapped reference. It keeps tracebacks, help output, and introspection useful. Prefer it in almost every wrapper.

Use try/finally when cleanup or measurement must happen even if the wrapped function raises. The wrapper should normally return the wrapped function's result unchanged and let its exceptions propagate unless changing that behavior is intentional.

Decorators with arguments

A parameterized decorator adds an outer factory call:

from functools import wraps

def repeat(times):
    def decorate(function):
        @wraps(function)
        def wrapper(*args, **kwargs):
            result = None
            for _ in range(times):
                result = function(*args, **kwargs)
            return result
        return wrapper
    return decorate

@repeat(3)
def greet(name):
    print(f"Hello, {name}!")

This is equivalent to greet = repeat(3)(greet). The three layers have distinct jobs: accept configuration, accept the target function, then handle calls.

A decorator that supports both @decorator and @decorator(option=...) needs to distinguish the two forms. Avoid adding that complexity unless callers need both spellings.

Useful functools tools

  • @functools.wraps(original) preserves the wrapped callable's metadata.
  • functools.lru_cache(maxsize=128) memoizes calls using hashable arguments. It is appropriate for deterministic functions; it retains results and should not cache side effects or changing external data.
  • functools.cache is an unbounded cache (Python 3.9+); use it only when the key space and lifetime are controlled.
  • functools.partial(function, fixed_arg=...) creates a callable with some arguments pre-filled.
  • functools.singledispatch dispatches a function based on the type of its first argument.
  • functools.cached_property computes an instance property once and stores it on that instance (Python 3.8+).

Example:

from functools import lru_cache

@lru_cache(maxsize=256)
def fibonacci(n: int) -> int:
    if n < 2:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

Cached arguments must be hashable. Be deliberate about cache invalidation and shared state.

3. Class-based decorators

A class can act as a decorator by accepting the target callable in init and implementing call:

from functools import update_wrapper

class CountCalls:
    def __init__(self, function):
        self.function = function
        self.calls = 0
        update_wrapper(self, function)

    def __call__(self, *args, **kwargs):
        self.calls += 1
        return self.function(*args, **kwargs)

@CountCalls
def square(value):
    return value * value

assert square(4) == 16
assert square.calls == 1

update_wrapper serves the same metadata-preservation role as @wraps. A class-based decorator is handy when the decorator itself needs inspectable state or methods. For ordinary wrapping, a closure is often simpler.

Methods caveat: A callable decorator object is not automatically a descriptor. When a decorated function is stored on a class, method binding may not behave as expected. For a decorator intended to support instance methods, implement get (often by returning types.MethodType(self, instance)) or use a function wrapper. Include a method-binding example in tests when supporting methods is part of the contract.

4. Context managers and contextlib

A context manager brackets work with setup and cleanup. The with statement guarantees the exit protocol runs even if the body raises.

class ManagedFile:
    def __init__(self, path, mode="r", encoding="utf-8"):
        self.path = path
        self.mode = mode
        self.encoding = encoding

    def __enter__(self):
        self.file = open(self.path, self.mode, encoding=self.encoding)
        return self.file

    def __exit__(self, exc_type, exc_value, traceback):
        self.file.close()
        return False  # Do not suppress exceptions.

Prefer the standard library's file manager in real code: with open(path, encoding="utf-8") as file:. In custom exit, return a truthy value only when intentionally suppressing an exception.

contextlib provides concise building blocks:

from contextlib import contextmanager
from time import perf_counter

@contextmanager
def elapsed(label):
    started = perf_counter()
    try:
        yield
    finally:
        print(f"{label}: {perf_counter() - started:.3f}s")

with elapsed("work"):
    sum(range(1_000_000))

The code before yield is enter logic; the code after it is exit logic. finally ensures cleanup happens on normal completion and exceptions.

Other useful tools include:

  • contextlib.suppress(SomeError) for a narrowly scoped exception that is safe to ignore.
  • contextlib.ExitStack when the number of context managers is dynamic.
  • contextlib.closing(obj) for an object with close() but no context-manager protocol.
  • contextlib.nullcontext(value) as a no-op context manager, often useful in conditional code.

5. Descriptors: controlling attribute access

A descriptor is an object defining one or more of get, set, or delete. When stored as a class attribute, Python calls these methods during attribute access. Properties are descriptors; custom descriptors can reuse validation behavior across many fields.

class Positive:
    def __set_name__(self, owner, name):
        self.name = name
        self.storage_name = "_" + name

    def __get__(self, instance, owner=None):
        if instance is None:
            return self
        return getattr(instance, self.storage_name, 0)

    def __set__(self, instance, value):
        if value <= 0:
            raise ValueError(f"{self.name} must be positive")
        setattr(instance, self.storage_name, value)

class Product:
    quantity = Positive()

    def __init__(self, quantity):
        self.quantity = quantity

item = Product(4)
assert item.quantity == 4

__set_name__ is called when the owning class is created. A descriptor that defines set is a data descriptor and generally takes precedence over an instance attribute of the same name. A descriptor defining only get is a non-data descriptor and can be shadowed by an instance attribute.

Descriptors are powerful, but they add indirection. Use a property for one-off validation; use a descriptor when the same attribute behavior is shared across several classes or fields.

6. Choosing a class extension mechanism

Python offers several ways to customize class behavior. Pick the least powerful tool that expresses the requirement clearly.

__init_subclass__: a lightweight subclass hook

Python calls a base class's __init_subclass__ when a subclass is created. It can validate or register subclasses without introducing a metaclass.

class Plugin:
    registry = {}

    def __init_subclass__(cls, *, name=None, **kwargs):
        super().__init_subclass__(**kwargs)
        if name is not None:
            if name in Plugin.registry:
                raise ValueError(f"duplicate plugin name: {name}")
            Plugin.registry[name] = cls

class JsonPlugin(Plugin, name="json"):
    pass

Forward unknown class keywords through super().__init_subclass__(**kwargs) so cooperative multiple inheritance can work.

Metaclasses: customizing class creation

A metaclass is the class of a class. It controls how class objects are built; the default is type. Metaclasses can inspect or transform a class namespace before the class exists.

class RequireDocstring(type):
    def __new__(mcls, name, bases, namespace):
        if name != "Base" and not namespace.get("__doc__"):
            raise TypeError(f"{name} needs a docstring")
        return super().__new__(mcls, name, bases, namespace)

class Base(metaclass=RequireDocstring):
    """Base for documented subclasses."""

class Concrete(Base):
    """A documented concrete type."""

Metaclasses affect every subclass and can conflict when multiple bases have incompatible metaclasses. Use one when class creation itself must be controlled and __init_subclass__ or a class decorator cannot do the job. For simple registration, validation, or configuration, start with __init_subclass__.

Dataclasses: less boilerplate for data

dataclasses generates common methods from annotated fields:

from dataclasses import dataclass, field

@dataclass
class Job:
    name: str
    retries: int = 3
    tags: list[str] = field(default_factory=list)

job = Job("thumbnail")
job.tags.append("image")

Use default_factory for mutable defaults so each instance gets its own list. frozen=True makes attribute reassignment unavailable through the normal interface; it does not recursively freeze objects stored in fields. Dataclasses are a good fit for data containers, not a substitute for every class.

Protocols and ABCs: describing interfaces

A Protocol supports structural typing: an object is compatible if it has the required shape, whether or not it explicitly inherits from the protocol.

from typing import Protocol

class SupportsClose(Protocol):
    def close(self) -> None: ...

def shut_down(resource: SupportsClose) -> None:
    resource.close()

An abstract base class (ABC) supports nominal relationships and can prevent instantiation until required methods are implemented:

from abc import ABC, abstractmethod

class Serializer(ABC):
    @abstractmethod
    def dumps(self, value: object) -> str:
        """Serialize a value."""

class JsonSerializer(Serializer):
    def dumps(self, value: object) -> str:
        import json
        return json.dumps(value)

Use a protocol when callers care about capabilities and third-party types should fit without inheritance. Use an ABC when a shared base implementation, explicit runtime family, or enforced subclass contract is useful. Static type checkers understand both; runtime checking of protocols requires @runtime_checkable and only verifies limited structural presence, not full method signatures.

7. Lab: a retry, caching, and timing decorator library

Build a small module named decorators.py. It should offer three configurable decorators:

  • @retry(attempts=3, delay=0.0, exceptions=(Exception,)): retry selected failures, then re-raise the final exception.
  • @cache(maxsize=128): cache successful return values by arguments.
  • @timed: measure each actual invocation and report elapsed time.

This implementation is intentionally small and synchronous. It does not provide async support, cache invalidation, jitter, or distributed coordination.

# decorators.py
from collections.abc import Callable
from functools import lru_cache, wraps
from time import perf_counter, sleep
from typing import ParamSpec, TypeVar

P = ParamSpec("P")
R = TypeVar("R")


def retry(
    attempts: int = 3,
    *,
    delay: float = 0.0,
    exceptions: tuple[type[Exception], ...] = (Exception,),
) -> Callable[[Callable[P, R]], Callable[P, R]]:
    if attempts < 1:
        raise ValueError("attempts must be at least 1")
    if delay < 0:
        raise ValueError("delay cannot be negative")
    if not exceptions:
        raise ValueError("exceptions cannot be empty")

    def decorate(function: Callable[P, R]) -> Callable[P, R]:
        @wraps(function)
        def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
            for attempt in range(1, attempts + 1):
                try:
                    return function(*args, **kwargs)
                except exceptions:
                    if attempt == attempts:
                        raise
                    if delay:
                        sleep(delay)
            raise AssertionError("unreachable")
        return wrapper
    return decorate


def cache(maxsize: int = 128):
    if maxsize < 1:
        raise ValueError("maxsize must be at least 1")

    def decorate(function):
        cached = lru_cache(maxsize=maxsize)(function)
        @wraps(function)
        def wrapper(*args, **kwargs):
            return cached(*args, **kwargs)
        wrapper.cache_info = cached.cache_info
        wrapper.cache_clear = cached.cache_clear
        return wrapper
    return decorate


def timed(function: Callable[P, R]) -> Callable[P, R]:
    @wraps(function)
    def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
        started = perf_counter()
        try:
            return function(*args, **kwargs)
        finally:
            elapsed = perf_counter() - started
            print(f"{function.__qualname__}: {elapsed:.6f}s")
    return wrapper

ParamSpec carries a wrapped function's parameter types through the decorator, and TypeVar carries its return type. The runtime behavior works without static typing; these annotations help type checkers and editors.

Compose the decorators

from decorators import cache, retry, timed

calls = 0

@timed
@cache(maxsize=32)
@retry(attempts=3, delay=0.05, exceptions=(TimeoutError,))
def fetch_record(record_id: int) -> dict[str, int]:
    global calls
    calls += 1
    if calls == 1:
        raise TimeoutError("temporary failure")
    return {"id": record_id}

Decorators apply from the bottom up. Here, retry wraps the original function, cache wraps the retry wrapper, and timed wraps the cached callable. A repeated successful call with the same key is served from cache, so the outer timer measures the cache lookup on a hit. To time only actual computation, put @timed below @cache instead. To retry cache lookups as well, choose the order deliberately and document it.

The global above is included only to make the example easy to read; production code should usually keep state in an object or use a closure. Avoid caching mutable results if callers can mutate them, and remember that lru_cache requires hashable arguments. Exceptions are not cached by lru_cache.

Lab tasks

  1. Put the implementation in decorators.py and add a short module docstring describing the scope and limitations.
  2. Demonstrate that retry calls again after a TimeoutError, returns on success, and re-raises after the final attempt.
  3. Demonstrate that exceptions=(TimeoutError,) does not retry a ValueError.
  4. Call a cached function twice with the same hashable arguments; inspect cache_info() and clear it with cache_clear().
  5. Verify that the timer reports elapsed time when the wrapped function returns and when it raises.
  6. Compose decorators in at least two orders and explain what the timer measures and what gets cached.
  7. Preserve name, doc, and the signature metadata as far as possible with wraps.
  8. Add a class-based Retry decorator as an extension. Decide explicitly whether it supports instance methods; if it does, implement and demonstrate descriptor binding.

Design questions

  • Should attempts mean total calls or retries after the first call? This tutorial defines it as total calls.
  • Should delays be fixed or exponential? This lab uses a fixed delay; a production implementation may add bounded exponential backoff and jitter.
  • Should timing print, log, or call a supplied callback? Printing keeps the exercise self-contained; a library should usually accept a reporting hook or use logging.
  • Should a cache be shared across decorated functions or scoped to one function? This implementation creates one cache per decorated function.

8. Suggested practice and review

Try these exercises after the lab:

  1. Write a closure that assigns monotonically increasing IDs and explain where its state lives.
  2. Write a decorator factory that rejects calls when a predicate is false, preserving metadata and exceptions.
  3. Implement a Range descriptor that validates an inclusive lower and upper bound, then use it in two classes.
  4. Use ExitStack to open a variable number of files safely.
  5. Register plugin subclasses with __init_subclass__; then describe a requirement that would actually justify a metaclass.
  6. Model a Task record with a dataclass and type a consumer against a Protocol.

Key takeaways

  • A closure retains access to its enclosing scope; nonlocal supports rebinding captured state.
  • A decorator is callable transformation. Parameterized decorators add a factory layer.
  • wraps and update_wrapper preserve useful introspection metadata.
  • Context managers make setup and cleanup reliable, including on exceptions.
  • Descriptors control class-level attribute behavior; properties cover many simple cases.
  • Dataclasses model data, protocols describe structural capabilities, ABCs define nominal contracts, and __init_subclass__ handles many class-registration needs.
  • Metaclasses are a powerful class-creation hook and should be used only when simpler mechanisms cannot express the requirement.
  • Decorator order changes semantics. Make it explicit and demonstrate it.

Friday, November 6, 2026

Advanced Functions and OOP

1. Closures in Python

1.1 What is a Closure?

A closure is a function that remembers variables from its enclosing scope, even after the outer function has finished executing.

Closures are useful for:

  • Maintaining state without global variables.

  • Creating configurable functions.

  • Implementing decorators.

  • Building function factories.

1.2 Understanding Closures

Consider this example:

def outer_function(message):

    def inner_function():
        print(message)

    return inner_function


my_function = outer_function("Hello Python!")

my_function()

Output:

Hello Python!

Explanation:

  1. outer_function() receives a message.

  2. It defines an inner function.

  3. The inner function uses the outer function's variable.

  4. The outer function returns the inner function.

  5. Even after outer_function() finishes, the inner function remembers message.

That is a closure.

1.3 Closure with State

def counter():

    count = 0

    def increment():

        nonlocal count
        count += 1

        return count

    return increment


my_counter = counter()

print(my_counter())
print(my_counter())
print(my_counter())

Output:

1
2
3

Important concept: nonlocal allows an inner function to modify a variable defined in its enclosing function.

Without nonlocal, Python treats an assignment to count as a new local variable.

1.4 Practical Example: Bank Account

def create_account(initial_balance):

    balance = initial_balance

    def deposit(amount):
        nonlocal balance
        balance += amount
        return balance

    def withdraw(amount):
        nonlocal balance

        if amount <= balance:
            balance -= amount
        else:
            print("Insufficient balance")

        return balance

    return deposit, withdraw


deposit, withdraw = create_account(1000)

print(deposit(500))
print(withdraw(200))

Output:

1500
1300

Interview question: What is the difference between a closure and a global variable?

Answer: A closure maintains access to variables in its enclosing lexical scope. A global variable belongs to the module-level namespace and can be accessed by functions throughout that module.


2. Decorators in Python

2.1 What is a Decorator?

A decorator is a function that modifies or extends the behavior of another function without changing its original source code.

Decorators are commonly used for:

  • Logging

  • Authentication

  • Authorization

  • Performance monitoring

  • Caching

  • Retry mechanisms

  • Input validation

2.2 Basic Decorator

def my_decorator(func):

    def wrapper():
        print("Before function execution")

        func()

        print("After function execution")

    return wrapper


@my_decorator
def say_hello():
    print("Hello World!")


say_hello()

Output:

Before function execution
Hello World!
After function execution

The following:

@my_decorator
def say_hello():
    print("Hello World!")

is equivalent to:

def say_hello():
    print("Hello World!")

say_hello = my_decorator(say_hello)

2.3 Decorators with Arguments

A decorator should support functions that accept different parameters.

def my_decorator(func):

    def wrapper(*args, **kwargs):

        print("Function started")

        result = func(*args, **kwargs)

        print("Function completed")

        return result

    return wrapper


@my_decorator
def add(a, b):
    return a + b


print(add(10, 20))

Output:

Function started
Function completed
30

Important:

  • *args accepts positional arguments.

  • **kwargs accepts keyword arguments.

  • return result preserves the original function's return value.

2.4 Decorators with Their Own Arguments

Sometimes we want to configure the decorator itself.

Example: Repeat a function a specified number of times.

def repeat(times):

    def decorator(func):

        def wrapper(*args, **kwargs):

            for _ in range(times):
                result = func(*args, **kwargs)

            return result

        return wrapper

    return decorator


@repeat(3)
def greet():
    print("Hello!")


greet()

Output:

Hello!
Hello!
Hello!

Notice the three levels:

  1. repeat(times) receives decorator configuration.

  2. decorator(func) receives the original function.

  3. wrapper(*args, **kwargs) executes the function.

2.5 Preserve Function Metadata Using functools.wraps

Decorators can overwrite the original function's name and documentation.

import functools

def my_decorator(func):

    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        return func(*args, **kwargs)

    return wrapper


@my_decorator
def calculate():
    """Calculate something."""
    return 100


print(calculate.__name__)
print(calculate.__doc__)

Output:

calculate
Calculate something.

Always consider using functools.wraps when writing decorators.

2.6 Class-Based Decorators

Python also allows us to implement decorators using classes.

A class-based decorator implements the __call__() method.

import functools

class Logger:

    def __init__(self, func):
        functools.update_wrapper(self, func)
        self.func = func

    def __call__(self, *args, **kwargs):

        print(f"Calling {self.func.__name__}")

        result = self.func(*args, **kwargs)

        print("Execution completed")

        return result


@Logger
def add(a, b):
    return a + b


print(add(10, 20))

Output:

Calling add
Execution completed
30

The class instance behaves like a function because it implements __call__().

2.7 Class-Based Decorator with Arguments

import functools

class Repeat:

    def __init__(self, times):
        self.times = times

    def __call__(self, func):

        @functools.wraps(func)
        def wrapper(*args, **kwargs):

            result = None

            for _ in range(self.times):
                result = func(*args, **kwargs)

            return result

        return wrapper


@Repeat(3)
def greet():
    print("Welcome!")


greet()

This approach is useful when a decorator requires configuration or maintains internal state.


3. The functools Module

The functools module provides tools for working with higher-order functions and callable objects.

3.1 functools.wraps()

Preserves the metadata of the original function.

from functools import wraps

Use it when creating custom decorators.

3.2 functools.lru_cache()

Caches the result of function calls.

from functools import lru_cache

@lru_cache(maxsize=128)
def fibonacci(n):

    if n < 2:
        return n

    return fibonacci(n - 1) + fibonacci(n - 2)


print(fibonacci(10))

print(fibonacci.cache_info())

Output:

55
CacheInfo(hits=8, misses=11, maxsize=128, currsize=11)

Benefits:

  • Reduces repeated calculations.

  • Improves performance.

  • Useful for recursive functions.

The arguments must be hashable, and cached results should be safe to reuse.

3.3 functools.cache()

An unbounded caching decorator.

from functools import cache

@cache
def square(n):
    print("Calculating...")
    return n * n


print(square(5))
print(square(5))

The calculation runs only once for the same argument.

3.4 functools.partial()

Creates a new function by fixing some arguments of an existing function.

from functools import partial

def multiply(a, b):
    return a * b


double = partial(multiply, 2)

print(double(10))
print(double(20))

Output:

20
40

3.5 functools.singledispatch()

Allows a function to have different implementations based on the type of its first argument.

from functools import singledispatch

@singledispatch
def display(value):
    print("Default:", value)


@display.register
def _(value: int):
    print("Integer:", value)


@display.register
def _(value: list):
    print("List:", value)


display(100)
display([1, 2, 3])
display("Python")

This is useful when implementing type-specific behavior.


4. Context Managers and contextlib

4.1 What is a Context Manager?

A context manager manages resources and ensures that cleanup operations happen correctly.

Common examples:

  • Opening and closing files.

  • Managing database connections.

  • Managing locks.

  • Handling transactions.

  • Managing temporary resources.

Python provides the with statement for context managers.

4.2 File Handling Example

with open("sample.txt", "w") as file:
    file.write("Learning Python")

The file is automatically closed when the with block exits, including when an exception occurs.

4.3 Creating a Custom Context Manager

A context manager class implements:

  • __enter__()

  • __exit__()

class MyContext:

    def __enter__(self):
        print("Entering context")
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        print("Exiting context")


with MyContext():
    print("Inside context")

Output:

Entering context
Inside context
Exiting context

The __exit__() method receives exception information.

Returning True from __exit__() suppresses the exception. Returning False or None allows it to propagate.

4.4 Using contextlib.contextmanager

The contextlib module provides a simpler way to create context managers using generators.

from contextlib import contextmanager

@contextmanager
def my_context():

    print("Starting")

    try:
        yield
    finally:
        print("Cleaning up")


with my_context():
    print("Executing task")

Output:

Starting
Executing task
Cleaning up

The yield statement separates setup from the managed block and cleanup.

4.5 Other Useful contextlib Features

closing(): Ensures that an object with a close() method is closed.

from contextlib import closing

with closing(open("sample.txt")) as file:
    print(file.read())

suppress(): Suppresses specified exceptions.

from contextlib import suppress

with suppress(FileNotFoundError):
    open("missing.txt")

ExitStack: Manages multiple context managers dynamically.

from contextlib import ExitStack

with ExitStack() as stack:

    files = [
        stack.enter_context(open("a.txt")),
        stack.enter_context(open("b.txt"))
    ]

    print("Both files are open")

5. Descriptors in Python

5.1 What is a Descriptor?

A descriptor is an object that defines one or more of these methods:

  • __get__()

  • __set__()

  • __delete__()

Descriptors control how attributes are accessed, assigned, and deleted.

They are used internally by properties, methods, and many Python frameworks.

5.2 Basic Descriptor Example

class Age:

    def __get__(self, instance, owner):
        return instance._age

    def __set__(self, instance, value):

        if value < 0:
            raise ValueError("Age cannot be negative")

        instance._age = value


class Person:

    age = Age()


person = Person()

person.age = 25

print(person.age)

Output:

25

Trying to assign a negative age raises a ValueError.

5.3 Data and Non-Data Descriptors

A data descriptor implements __set__() or __delete__().

A non-data descriptor implements only __get__().

Data descriptors generally take precedence over an instance's dictionary when resolving attributes.

5.4 Practical Uses

Descriptors are useful for:

  • Attribute validation.

  • Type checking.

  • Lazy loading.

  • Reusable properties.

  • ORM field management.

  • Framework-level attribute management.


6. Metaclasses in Python

6.1 What is a Metaclass?

A metaclass is a class whose instances are classes.

In Python, the default metaclass is type.

class Employee:
    pass


print(type(Employee))

Output:

<class 'type'>

Think of it this way:

  • Objects are created from classes.

  • Classes are created by metaclasses.

6.2 Creating a Custom Metaclass

class MyMeta(type):

    def __new__(cls, name, bases, namespace):

        print(f"Creating class: {name}")

        return super().__new__(
            cls, name, bases, namespace
        )


class Employee(metaclass=MyMeta):
    pass

Output:

Creating class: Employee

The __new__() method participates in class creation.

6.3 Practical Example: Automatically Register Classes

class PluginMeta(type):

    registry = {}

    def __new__(cls, name, bases, namespace):

        new_class = super().__new__(
            cls, name, bases, namespace
        )

        if name != "Plugin":
            cls.registry[name] = new_class

        return new_class


class Plugin(metaclass=PluginMeta):
    pass


class EmailPlugin(Plugin):
    pass


class PaymentPlugin(Plugin):
    pass


print(PluginMeta.registry)

This pattern is useful for plugin architectures and framework registration.

Important: Metaclasses are powerful but should be used only when simpler alternatives are insufficient.


7. init_subclass()

__init_subclass__() is called whenever a new subclass is created.

It provides a simpler alternative to many metaclass use cases.

Example

class Plugin:

    registry = []

    def __init_subclass__(cls, **kwargs):

        super().__init_subclass__(**kwargs)

        Plugin.registry.append(cls)


class EmailPlugin(Plugin):
    pass


class PaymentPlugin(Plugin):
    pass


print(Plugin.registry)

This automatically registers subclasses.

When should you use it?

Use __init_subclass__() when you want to:

  • Validate subclasses.

  • Register implementations.

  • Enforce class conventions.

  • Initialize subclass-level configuration.

Prefer it over a metaclass when you only need to customize subclass creation.


8. Dataclasses

8.1 What is a Dataclass?

A dataclass simplifies the creation of classes primarily used to store data.

The @dataclass decorator automatically generates methods such as:

  • __init__()

  • __repr__()

  • __eq__()

Example Without Dataclass

class Employee:

    def __init__(self, name, salary):
        self.name = name
        self.salary = salary

Same Example Using Dataclass

from dataclasses import dataclass

@dataclass
class Employee:

    name: str
    salary: float


emp = Employee("Rahul", 50000)

print(emp)

Output:

Employee(name='Rahul', salary=50000)

8.2 Default Values

from dataclasses import dataclass

@dataclass
class Product:

    name: str
    price: float
    quantity: int = 1


product = Product("Laptop", 50000)

print(product)

8.3 default_factory

Use default_factory for mutable default values.

from dataclasses import dataclass, field

@dataclass
class Student:

    name: str
    subjects: list = field(default_factory=list)


s1 = Student("Amit")
s2 = Student("Neha")

s1.subjects.append("Python")

print(s1.subjects)
print(s2.subjects)

Output:

['Python']
[]

Each object receives its own list.

8.4 Frozen Dataclasses

from dataclasses import dataclass

@dataclass(frozen=True)
class Configuration:

    host: str
    port: int


config = Configuration("localhost", 8000)

A frozen dataclass prevents normal attribute reassignment after initialization.


9. Protocols and Abstract Base Classes

These concepts help build flexible and maintainable application architectures.

9.1 Abstract Base Classes (ABCs)

An ABC defines a contract that subclasses must implement.

from abc import ABC, abstractmethod

class PaymentProcessor(ABC):

    @abstractmethod
    def process_payment(self, amount):
        pass


class CreditCardPayment(PaymentProcessor):

    def process_payment(self, amount):
        print(f"Processing payment: {amount}")


payment = CreditCardPayment()

payment.process_payment(1000)

A subclass that does not implement all abstract methods cannot be instantiated.

9.2 Why Use ABC?

Use ABC when:

  • You want explicit inheritance.

  • You want to enforce implementation requirements.

  • You need shared behavior.

  • You are designing a framework or application architecture.

9.3 Protocol

A Protocol supports structural subtyping.

This means a class can satisfy an interface by implementing the required methods, without explicitly inheriting from that interface.

from typing import Protocol

class PaymentProcessor(Protocol):

    def process_payment(self, amount: float) -> None:
        ...


class CreditCardPayment:

    def process_payment(self, amount: float) -> None:
        print(f"Paid {amount}")


def checkout(
    processor: PaymentProcessor,
    amount: float
):
    processor.process_payment(amount)


checkout(CreditCardPayment(), 500)

The CreditCardPayment class does not need to inherit from PaymentProcessor.

9.4 ABC vs Protocol

FeatureABCProtocol
Typing approachNominalStructural
Explicit inheritanceUsually requiredNot required
Abstract methodsYesMethod signatures
Runtime enforcementYes, for abstract methodsNot by default
Main purposeEnforce contractsFlexible interfaces

Use ABC when inheritance and runtime enforcement are important.

Use Protocol when you want flexible interfaces and better static type checking.


10. LAB PROJECT: Build a Retry, Caching and Timing Decorator Library

Project Objective

Build a reusable Python library that provides three decorators:

  1. @retry – Automatically retries failed function calls.

  2. @cache_result – Caches function results.

  3. @timing – Measures function execution time.

These decorators should work with ordinary functions and support function arguments.

Project Structure

week2_project/
│
├── decorators.py
├── main.py
└── requirements.txt

We will use Python's standard library, so no external packages are required.

Step 1: Implement Timing Decorator

Create decorators.py.

import functools
import time


def timing(func):

    @functools.wraps(func)
    def wrapper(*args, **kwargs):

        start = time.perf_counter()

        try:
            return func(*args, **kwargs)

        finally:
            elapsed = time.perf_counter() - start

            print(
                f"{func.__name__} executed in "
                f"{elapsed:.4f} seconds"
            )

    return wrapper

Explanation:

  • time.perf_counter() measures elapsed time.

  • functools.wraps() preserves function metadata.

  • finally ensures timing is printed even if the function raises an exception.

Step 2: Implement Retry Decorator

Add the following code to decorators.py.

def retry(max_attempts=3, delay=1, exceptions=(Exception,)):

    def decorator(func):

        @functools.wraps(func)
        def wrapper(*args, **kwargs):

            for attempt in range(1, max_attempts + 1):

                try:
                    return func(*args, **kwargs)

                except exceptions:

                    if attempt == max_attempts:
                        raise

                    print(
                        f"Attempt {attempt} failed. "
                        f"Retrying..."
                    )

                    time.sleep(delay)

        return wrapper

    return decorator

Usage:

@retry(max_attempts=3, delay=2)
def connect_api():

    print("Connecting to API")

    raise ConnectionError("Server unavailable")


connect_api()

The function will execute three times before the final exception is raised.

Important production considerations:

  • Retry only transient failures.

  • Use exponential backoff for real services.

  • Avoid retrying non-idempotent operations unless they are designed to be safe to retry.

  • Consider adding jitter and maximum delay.

Step 3: Implement Caching Decorator

Add this code to decorators.py.

def cache_result(maxsize=128):

    def decorator(func):

        cached_func = functools.lru_cache(
            maxsize=maxsize
        )(func)

        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            return cached_func(*args, **kwargs)

        wrapper.cache_info = cached_func.cache_info
        wrapper.cache_clear = cached_func.cache_clear

        return wrapper

    return decorator

Usage:

@cache_result(maxsize=100)
def calculate_square(number):

    print("Calculating...")

    return number * number


print(calculate_square(10))
print(calculate_square(10))

print(calculate_square.cache_info())

The second call reuses the cached result.

Note: The cache uses function arguments as keys. Arguments must be hashable. Cached functions should generally avoid side effects, and cache invalidation should be considered when underlying data changes.

Step 4: Use All Three Decorators Together

Create main.py.

from decorators import timing, retry, cache_result

@timing
@retry(max_attempts=3, delay=1)
@cache_result(maxsize=100)
def fetch_data(user_id):

    print(f"Fetching data for user {user_id}")

    return {
        "user_id": user_id,
        "name": "Rahul"
    }


print(fetch_data(101))

print(fetch_data(101))

The decorators are applied from bottom to top:

  1. cache_result wraps the original function.

  2. retry wraps the cached function.

  3. timing wraps the retry-enabled function.

The execution flow is important because decorator order changes behavior.

Step 5: Test Your Library

Test the following scenarios:

  1. A function that succeeds on the first attempt.

  2. A function that fails twice and succeeds on the third attempt.

  3. A function that always fails.

  4. A function that returns the same result for repeated arguments.

  5. A function that receives keyword arguments.

  6. A function that takes several seconds to execute.

  7. Two different functions using the same decorators.

Suggested testing framework: pytest.


11. Additional Advanced Challenges

Once the main project is complete, try these extensions:

Challenge 1: Exponential Backoff

Instead of a fixed delay, implement:

delay = initial_delay * (2 ** (attempt - 1))

Challenge 2: Cache Expiration

Add a TTL (Time To Live) option so cached results expire after a specified number of seconds.

Challenge 3: Class-Based Timing Decorator

Implement timing using a class with __init__() and __call__().

Challenge 4: Logging Decorator

Create a decorator that logs:

  • Function name

  • Input arguments

  • Execution time

  • Return value

  • Exceptions

Avoid logging passwords, tokens, or other sensitive arguments.

Challenge 5: Async Retry Decorator

Extend the retry decorator to support async def functions using asyncio.sleep().


12. Week 2 Interview Questions

  1. What is a closure in Python?

  2. What is the purpose of nonlocal?

  3. What is a decorator?

  4. What is the difference between a decorator and a closure?

  5. What is the purpose of functools.wraps()?

  6. How do decorators with arguments work?

  7. How do class-based decorators work?

  8. What is the difference between cache and lru_cache?

  9. What is functools.partial()?

  10. What is a context manager?

  11. What are __enter__() and __exit__()?

  12. What is the purpose of contextlib?

  13. What is a descriptor?

  14. What is the difference between data and non-data descriptors?

  15. What is a metaclass?

  16. What is the default metaclass in Python?

  17. What is the purpose of __init_subclass__()?

  18. What is a dataclass?

  19. What is the purpose of default_factory?

  20. What is a frozen dataclass?

  21. What is an Abstract Base Class?

  22. What is a Protocol?

  23. What is the difference between ABC and Protocol?

  24. What happens when multiple decorators are applied to a function?

  25. How would you implement a retry decorator for an API?


13. Week 2 Completion Checklist

  • Understand closures and lexical scoping.

  • Implement decorators with and without arguments.

  • Implement class-based decorators.

  • Use functools.wraps, lru_cache, cache, and partial.

  • Create custom context managers.

  • Understand descriptors.

  • Understand metaclasses and __init_subclass__.

  • Implement dataclasses.

  • Understand ABCs and Protocols.
  • Complete the retry decorator.

  • Complete the caching decorator.

  • Complete the timing decorator.

  • Test the library using pytest.

Final Outcome

At the end of Week 2, you should be able to design reusable Python components and understand many of the advanced mechanisms used internally by professional Python frameworks and libraries.

Recommended hands-on study sequence

Day

Topic

Practical assignment

Day 1

Closures

Build a counter and bank account

Day 2

Decorators

Logging and authentication decorators

Day 3

functools

Fibonacci caching and partial functions

Day 4

Context managers

Database connection manager

Day 5

Descriptors and metaclasses

Attribute validation and plugin registry

Day 6

Dataclasses, ABC, Protocol

Payment processing architecture

Day 7

Lab project

Build and test the decorator library


Week 2 Tutorial: Advanced Functions and OOP

Overview

By the end of this tutorial, you will be able to write decorators and context managers confidently, control attribute access with descriptors, customize class creation, and choose the right tool for modeling data and interfaces.

This course walks you through: functions and classes are objects (names bound to them), and dunder methods are the hooks Python calls for you. Every example runs on Python 3.11 or newer; outputs shown were produced on Python 3.13.

Part

Topic

You will be able to

1

Closures

Explain how a function keeps variables alive, and avoid the late-binding trap

2

Decorators and functools

Write plain, parameterized, and class-based decorators that preserve metadata

3

Context managers

Write them as classes and with contextlib, and combine many with ExitStack

4

Descriptors

Build reusable validated attributes and explain how property and methods work

5

Metaclasses and __init_subclass__

Explain how a class is created, and build a plugin registry without a metaclass

6

Dataclasses

Declare data classes with defaults, ordering, immutability, and slots

7

Protocol and ABCs

Pick between nominal and structural typing for an interface

Lab

Decorator library

Ship a tested timed, retry, and ttl_cache library

Prerequisites: Comfort with functions, classes, and exceptions.

Time: about 5 hours: roughly 2 hours of reading with a REPL open, and 3 hours for the lab and exercises.

1. Closures

A closure is a function that remembers variables from the scope where it was defined, even after that scope has finished running.

Functions are objects, so a function can be created inside another function and returned. The inner function keeps references to the outer variables it uses; those are called free variables.

def make_multiplier(n):
    def multiply(x):
        return x * n      # n is a free variable, taken from make_multiplier
    return multiply

double = make_multiplier(2)
triple = make_multiplier(3)
print(double(5), triple(5))   # 10 15

Each call to make_multiplier creates a fresh n and a fresh multiply, so double and triple do not interfere.

Looking inside a closure

The remembered variables live in cells, which you can inspect:

def make_counter():
    count = 0
    def counter():
        nonlocal count
        count += 1
        return count
    return counter

c1, c2 = make_counter(), make_counter()
print(c1(), c1(), c2())                  # 1 2 1: each closure has its own count
print(c1.__code__.co_freevars)           # ('count',)
print(c1.__closure__[0].cell_contents)   # 2

A closure is a lightweight alternative to a class with one method and a bit of state.

nonlocal: rebinding a free variable

Reading a free variable just works. Assigning to a name inside a function makes it local to that function, so rebinding an outer variable needs nonlocal.

def broken():
    total = 0
    def add(x):
        total += x     # total is treated as local here, and has no value yet
    return add

broken()(1)
# UnboundLocalError: cannot access local variable 'total' where it is not associated with a value

Adding nonlocal total inside add fixes it. Mutating an outer object (such as items.append(x)) never needs nonlocal, because it does not rebind the name; this is the mutate-versus-rebind distinction.

The late-binding trap

A closure looks up its free variables when it is called, not when it is defined. In a loop, every closure then sees the variable's final value.

funcs = [lambda: i for i in range(3)]
print([f() for f in funcs])     # [2, 2, 2]: all three share one i

Three ways to capture the current value:

funcs = [lambda i=i: i for i in range(3)]           # default argument is evaluated now
print([f() for f in funcs])                         # [0, 1, 2]

from functools import partial
funcs = [partial(lambda i: i, i) for i in range(3)]   # partial stores the argument now
print([f() for f in funcs])                         # [0, 1, 2]

def make(i):                                        # a factory function gets its own scope
    return lambda: i
funcs = [make(i) for i in range(3)]                 # [0, 1, 2]

Closures are the mechanism behind decorators, which come next: a decorator is a function that returns a closure around the function it received.

2. Decorators and functools

A decorator is a function that takes a function and returns a replacement for it, and the @ line is shorthand for that replacement.

@decorator
def f(): ...

# is exactly the same as
def f(): ...
f = decorator(f)

Because the replacement is usually a closure (section 1) that calls the original, a decorator can add behavior before and after a call without changing the function's code.

A first decorator

import functools
import time

def timed(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        try:
            return func(*args, **kwargs)
        finally:
            print(f"{func.__name__} took {time.perf_counter() - start:.2f}s")
    return wrapper

@timed
def slow_add(a, b):
    """Add two numbers."""
    time.sleep(0.01)
    return a + b

print(slow_add(1, 2))     # slow_add took 0.01s, then 3

The wrapper accepts *args, **kwargs so it works for any signature, returns whatever the function returns, and uses try/finally so the timing prints even if the function raises.

Always use functools.wraps

Without it, the decorated function silently becomes the wrapper, which breaks help text, debugging, and tools that read names or docstrings.

def no_wraps(func):
    def wrapper(*a, **k):
        return func(*a, **k)
    return wrapper

@no_wraps
def hello():
    """Say hi."""

print(hello.__name__, hello.__doc__)     # wrapper None

print(slow_add.__name__, slow_add.__doc__)   # slow_add Add two numbers.
print(slow_add.__wrapped__.__name__)         # slow_add: wraps keeps a link to the original

@functools.wraps(func) copies __name__, __doc__, __module__, and related attributes onto the wrapper, updates its __dict__, and sets __wrapped__.

Decorators with arguments

When a decorator takes settings, as in @repeat(3), you need one more layer: a function that takes the settings and returns the real decorator. repeat(3) is evaluated first, and the result is then applied to the function.

def repeat(times):                      # layer 1: takes the settings
    def decorator(func):                # layer 2: takes the function
        @functools.wraps(func)
        def wrapper(*a, **k):           # layer 3: runs on every call
            result = None
            for _ in range(times):
                result = func(*a, **k)
            return result
        return wrapper
    return decorator

@repeat(3)
def ping():
    print("ping")

ping()      # ping, ping, ping

@repeat(3) means ping = repeat(3)(ping). A frequent mistake is writing @repeat without parentheses; then repeat receives the function as times, and the failure appears later.

Supporting both @deco and @deco(...)

A small pattern lets one decorator work bare and with arguments, which is how the lab's timed works:

def logged(func=None, *, level="INFO"):
    def decorate(f):
        @functools.wraps(f)
        def wrapper(*a, **k):
            print(f"[{level}] calling {f.__name__}")
            return f(*a, **k)
        return wrapper
    return decorate if func is None else decorate(func)

@logged
def first(): return 1

@logged(level="DEBUG")
def second(): return 2

first(); second()
# [INFO] calling first
# [DEBUG] calling second

The bare form passes the function as func; the call form leaves func empty, so the decorator returns decorate for Python to apply. The * makes level keyword-only, which prevents a function from being mistaken for a setting.

Class-based decorators

Any callable can be a decorator. A class gives you somewhere natural to keep state: the instance replaces the function, and __call__ runs on every call.

class CountCalls:
    def __init__(self, func):
        functools.update_wrapper(self, func)   # like wraps, for an object
        self.func = func
        self.calls = 0

    def __call__(self, *args, **kwargs):
        self.calls += 1
        return self.func(*args, **kwargs)

@CountCalls
def say(msg):
    return msg

say("a"); say("b")
print(say.calls, say.__name__)    # 2 say

A class-based decorator has one trap: on a method it fails, because an instance of your class is not a function and so is not a descriptor (section 4); Python never binds self.

class Bad:
    @CountCalls
    def m(self): return "m"

Bad().m()    # TypeError: Bad.m() missing 1 required positional argument: 'self'

The fix is to give the decorator class a __get__ that returns a bound method:

import types

class CountCalls2(CountCalls):
    def __get__(self, instance, owner=None):
        if instance is None:
            return self                       # accessed on the class
        return types.MethodType(self, instance)

class Good:
    @CountCalls2
    def m(self): return "m"

g = Good()
print(g.m(), g.m(), Good.m.calls)    # m m 2

Prefer a function decorator when you do not need state on the decorator object itself; it works on methods with no extra code.

Stacking decorators

With more than one decorator, the one closest to the def wraps first, so @a over @b means f = a(b(f)). At call time the outermost wrapper runs first.

def tag(name):
    def deco(f):
        print(f"decorating with {name}")
        @functools.wraps(f)
        def wrapper(*a, **k):
            print(f"enter {name}")
            r = f(*a, **k)
            print(f"exit {name}")
            return r
        return wrapper
    return deco

@tag("outer")
@tag("inner")
def work():
    print("work")

work()


The left side of the drawing shows the wrapping, and the right side shows exactly what the code above prints, in order.

Order matters in practice. In this week's lab, ttl_cache sits closest to the function so that exceptions are never cached, retry wraps it, and timed is outermost so it measures every attempt together.

The functools toolbox

Tool

What it does

wraps, update_wrapper

Copy metadata from the wrapped function onto a wrapper

partial

Freeze some arguments of a function, producing a new callable

lru_cache, cache

Memoize results by arguments (cache is lru_cache(maxsize=None))

cached_property

Compute an attribute once per instance, then store it

singledispatch

Choose an implementation by the type of the first argument

reduce

Fold a sequence into one value with a two-argument function

total_ordering

Fill in comparison methods from __eq__ and one ordering method

import functools
from functools import partial

@functools.lru_cache(maxsize=None)
def fib(n):
    return n if n < 2 else fib(n - 1) + fib(n - 2)

print(fib(30))            # 832040, computed with only 31 distinct calls
print(fib.cache_info())   # CacheInfo(hits=28, misses=31, maxsize=None, currsize=31)
fib.cache_clear()         # empties the cache and resets the counters

int2 = partial(int, base=2)
print(int2("101"))        # 5

@functools.singledispatch
def describe(x):
    return f"object {x!r}"

@describe.register
def _(x: int): return f"int {x}"

@describe.register
def _(x: list): return f"list of {len(x)}"

print(describe(3), describe([1, 2]), describe("s"))   # int 3 list of 2 object 's'

class Circle:
    def __init__(self, r): self.r = r

    @functools.cached_property
    def area(self):
        print("computing")
        return 3.14159 * self.r ** 2

c = Circle(2)
print(c.area, c.area)     # computing, then 12.56636 12.56636

print(functools.reduce(lambda acc, x: acc * x, [1, 2, 3, 4], 1))   # 24

Two cautions for lru_cache: all arguments must be hashable, and on a method it keeps every self alive because self is part of the cache key. cached_property is a non-data descriptor that stores its result in the instance __dict__, as the descriptors section built by hand.

3. Context managers

A context manager guarantees that setup and cleanup code runs around a block, even if the block raises an exception.

The with statement is the tool: with open(path) as f: closes the file no matter how the block ends. Any object with __enter__ and __exit__ methods can be used this way.

Writing one as a class

import time

class Timer:
    def __enter__(self):
        self.start = time.perf_counter()
        return self                      # becomes the name after `as`

    def __exit__(self, exc_type, exc, tb):
        self.elapsed = time.perf_counter() - self.start
        return False                     # False: do not swallow exceptions

with Timer() as t:
    sum(range(10_000))
print(t.elapsed)

__exit__ receives the exception type, value, and traceback, or three None values when the block finished normally. Its return value matters:

  • Returning a falsy value (the default None or False) lets any exception continue propagating
  • Returning True suppresses the exception, and execution continues after the with block
class Swallow:
    def __enter__(self): return self
    def __exit__(self, exc_type, exc, tb):
        print("exit saw:", exc_type.__name__ if exc_type else None)
        return exc_type is ZeroDivisionError   # swallow only this one

with Swallow():
    1 / 0                     # exit saw: ZeroDivisionError
print("continued")            # reached: the error was suppressed

with Swallow():
    raise KeyError("k")       # exit saw: KeyError, then KeyError propagates

Suppress exceptions sparingly and only specific types; swallowing everything hides bugs.

Writing one with contextlib.contextmanager

For most cases a generator is shorter than a class. Code before yield is the setup, the yielded value goes to as, and code after yield is the cleanup.

from contextlib import contextmanager

@contextmanager
def tag(name):
    print(f"<{name}>")
    try:
        yield
    finally:
        print(f"</{name}>")

with tag("p"):
    print("hello")
# <p>
# hello
# </p>

An exception in the block is raised at the yield line. Always wrap the yield in try/finally when cleanup must run on errors; without it, the cleanup is skipped:

@contextmanager
def tag_no_finally(name):
    print(f"<{name}>")
    yield                      # an exception raised in the block surfaces here
    print(f"</{name}>")        # never reached when the block raises

A practical example that restores state:

import os

@contextmanager
def temp_chdir(path):
    old = os.getcwd()
    os.chdir(path)
    try:
        yield path
    finally:
        os.chdir(old)

Useful tools in contextlib

Tool

What it does

suppress(*exceptions)

Ignores the listed exceptions inside the block

redirect_stdout(f)

Sends print output to a file-like object temporarily

closing(obj)

Calls obj.close() on exit, for objects that have close but no __exit__

nullcontext(value)

Does nothing; a stand-in when a context manager is optional

ExitStack()

Manages a dynamic number of context managers and cleanup callbacks

ContextDecorator

Base class that lets a context manager also work as a decorator

asynccontextmanager

The async with version of contextmanager

import io
from contextlib import suppress, redirect_stdout, nullcontext

with suppress(FileNotFoundError):
    os.remove("/nonexistent/file")        # no error

buf = io.StringIO()
with redirect_stdout(buf):
    print("captured")
print(repr(buf.getvalue()))               # 'captured\n'

def maybe_lock(lock=None):
    return lock if lock is not None else nullcontext()

with maybe_lock():                        # works with or without a real lock
    print("no lock needed")

ExitStack: a variable number of resources

A with statement names its resources in the source code. When the number is only known at runtime, use ExitStack; it closes everything in reverse order, even if opening one of them fails.

from contextlib import ExitStack

paths = ["a.txt", "b.txt", "c.txt"]
with ExitStack() as stack:
    files = [stack.enter_context(open(p, "w")) for p in paths]
    stack.callback(print, "callback ran")   # arbitrary cleanup, runs last-in first-out
    for i, f in enumerate(files):
        f.write(str(i))
print([f.closed for f in files])            # [True, True, True]

One object, two roles: ContextDecorator

Subclass ContextDecorator and the same class works as a with block and as a function decorator. The library in this week's lab uses this for its Timer.

from contextlib import ContextDecorator

class Tracer(ContextDecorator):
    def __enter__(self):
        print("start"); return self
    def __exit__(self, *exc):
        print("stop"); return False

@Tracer()
def job():
    print("working")

job()      # start, working, stop

A function decorated with @contextmanager also gets this ability: tag("p") can be used both ways.

4. Descriptors

A descriptor is an object stored as a class attribute that controls what happens when an instance attribute is read, set, or deleted.

Descriptors are the machinery behind property, methods, classmethod, staticmethod, and __slots__. Learning them explains how Python attribute access really works.

A descriptor is any object whose class defines one or more of these methods:

Method

Called when

Signature

__get__

The attribute is read

__get__(self, obj, objtype=None)

__set__

The attribute is assigned on an instance

__set__(self, obj, value)

__delete__

The attribute is deleted with del

__delete__(self, obj)

__set_name__

The owning class is created

__set_name__(self, owner, name)

When __get__ is called on the class itself (Item.price), obj is None; convention is to return the descriptor in that case.

A reusable validated attribute

The descriptor lives on the class and is shared by all instances, so it must keep each instance's data on the instance. __set_name__ tells it the attribute name it was assigned to.

class Positive:
    def __set_name__(self, owner, name):
        self.private = "_" + name            # e.g. "_price"

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self                      # accessed on the class
        return getattr(obj, self.private)

    def __set__(self, obj, value):
        if value <= 0:
            raise ValueError(f"{self.private[1:]} must be positive, got {value}")
        setattr(obj, self.private, value)

class Item:
    price = Positive()
    qty = Positive()

    def __init__(self, price, qty):
        self.price = price                   # goes through Positive.__set__
        self.qty = qty

it = Item(9.5, 2)
print(it.price, it.qty)      # 9.5 2
print(vars(it))              # {'_price': 9.5, '_qty': 2}
it.qty = 0                   # ValueError: qty must be positive, got 0

One Positive class now validates any number of attributes in any number of classes, which is the advantage over writing a property pair for each one.

Do not store per-instance values on the descriptor itself (for example self.value = value): one descriptor object serves every instance, so all instances would share and overwrite that single value.

Data and non-data descriptors

Descriptors that define __set__ or __delete__ are data descriptors; those that define only __get__ are non-data descriptors. The difference decides which wins when the instance also has a value with the same name.

For obj.name, Python checks in this order:

  1. A data descriptor on the class (or its parents) wins
  2. Otherwise, an entry in the instance's __dict__
  3. Otherwise, a non-data descriptor or plain attribute on the class
  4. Otherwise, __getattr__ if the class defines it, or AttributeError
class DataD:
    def __get__(self, o, t=None): return "descriptor"
    def __set__(self, o, v): pass

class NonData:
    def __get__(self, o, t=None): return "descriptor"

class Demo:
    d = DataD()
    n = NonData()

dm = Demo()
dm.__dict__["d"] = "instance"
dm.__dict__["n"] = "instance"
print(dm.d, dm.n)      # descriptor instance

Functions are descriptors: how methods work

Every function defined in a class body is a non-data descriptor. Its __get__ returns a bound method that remembers the instance, which is why self is passed automatically.

class A:
    def hello(self): return "hi"

a = A()
print(type(A.__dict__["hello"]).__name__)    # function
bound = A.hello.__get__(a, A)                # what a.hello does behind the scenes
print(bound(), bound.__self__ is a)          # hi True

a.hello = lambda: "shadow"                   # allowed: non-data descriptors can be shadowed
print(a.hello())                             # shadow

This is also why a class-based decorator needs a __get__ method to work on methods, as the decorators section showed.

property is a data descriptor

class Temp:
    def __init__(self):
        self._c = 0

    @property
    def celsius(self):
        return self._c

    @celsius.setter
    def celsius(self, value):
        if value < -273.15:
            raise ValueError("below absolute zero")
        self._c = value

t = Temp(); t.celsius = 25
print(t.celsius)                             # 25
print(type(Temp.__dict__["celsius"]).__name__)   # property

Use property for one attribute in one class, and write your own descriptor when the same logic applies to many attributes.

A lazy attribute: a non-data descriptor that caches

Because a non-data descriptor loses to the instance __dict__, it can compute a value once, store it there, and never run again. This is how functools.cached_property works.

class Lazy:
    def __init__(self, func):
        self.func = func
        self.name = func.__name__

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        value = self.func(obj)
        obj.__dict__[self.name] = value     # shadows the descriptor from now on
        return value

class Report:
    @Lazy
    def data(self):
        print("loading...")
        return [1, 2, 3]

r = Report()
print(r.data)             # loading... then [1, 2, 3]
print(r.data)             # [1, 2, 3]: no second load
print("data" in vars(r))  # True

5. Metaclasses and __init_subclass__

A metaclass is the class of a class: it controls how a class object is created, the same way a class controls how its instances are created.

Classes are objects, so something must create them. By default that something is type.

print(type(42), type(int), type(type))    # <class 'int'> <class 'type'> <class 'type'>

type can also build a class directly from a name, a tuple of base classes, and a dictionary of attributes. A class statement is a convenient way to call it:

Point = type("Point", (), {"x": 0, "hello": lambda self: "hi"})
p = Point()
print(Point.__name__, p.x, p.hello())     # Point 0 hi

What happens when a class statement runs

Python runs these steps in order. The numbers are the order printed by the experiment below.

  1. The metaclass's __prepare__ returns the namespace dict for the class body
  2. The class body executes, filling that namespace
  3. The metaclass is called with the name, bases, and namespace: Meta.__new__ creates the class object
  4. Inside type.__new__, Python calls __set_name__ on every descriptor in the namespace, then __init_subclass__ on the parent class
  5. Meta.__init__ initializes the new class
  6. Later, calling the class to make an instance runs Meta.__call__, which in turn runs __new__ and __init__ of the class
class Meta(type):
    @classmethod
    def __prepare__(mcls, name, bases, **kw):
        print("1 __prepare__", name)
        return super().__prepare__(name, bases, **kw)
    def __new__(mcls, name, bases, ns, **kw):
        print("3 Meta.__new__", name)
        return super().__new__(mcls, name, bases, ns)
    def __init__(cls, name, bases, ns, **kw):
        print("5 Meta.__init__", name)
        super().__init__(name, bases, ns)
    def __call__(cls, *a, **k):
        print("6 Meta.__call__ (making an instance)")
        return super().__call__(*a, **k)

class Desc:
    def __set_name__(self, owner, name):
        print("4a __set_name__", name)

class Base:
    def __init_subclass__(cls, **kw):
        print("4b __init_subclass__", cls.__name__)
        super().__init_subclass__(**kw)

class Thing(Base, metaclass=Meta):
    print("2 class body runs")
    d = Desc()

obj = Thing()

Output, in order:

1 __prepare__ Thing
2 class body runs
3 Meta.__new__ Thing
4a __set_name__ d
4b __init_subclass__ Thing
5 Meta.__init__ Thing
6 Meta.__call__ (making an instance)

Example: a singleton metaclass

Overriding the metaclass's __call__ controls instance creation.

class Singleton(type):
    _instances = {}
    def __call__(cls, *a, **k):
        if cls not in cls._instances:
            cls._instances[cls] = super().__call__(*a, **k)
        return cls._instances[cls]

class Config(metaclass=Singleton):
    def __init__(self):
        print("init runs once")

print(Config() is Config())    # init runs once, then True

Prefer __init_subclass__ for most jobs

Metaclasses are powerful but heavy: a class can have only one metaclass, and combining libraries that each bring their own causes conflicts. Since Python 3.6, __init_subclass__ covers the most common reason to want one, which is reacting when someone subclasses your class.

class Plugin:
    registry = {}

    def __init_subclass__(cls, /, key=None, **kwargs):
        super().__init_subclass__(**kwargs)
        cls.registry[key or cls.__name__.lower()] = cls

class CsvPlugin(Plugin): ...
class JsonPlugin(Plugin, key="json"): ...

print(Plugin.registry)
# {'csvplugin': <class '__main__.CsvPlugin'>, 'json': <class '__main__.JsonPlugin'>}

The key keyword in the class line is passed straight to __init_subclass__. The same hook can enforce rules on subclasses:

class Task:
    required = ("run",)

    def __init_subclass__(cls, **kw):
        super().__init_subclass__(**kw)
        missing = [n for n in cls.required if not callable(getattr(cls, n, None))]
        if missing:
            raise TypeError(f"{cls.__name__} must define {missing}")

class Good(Task):
    def run(self): ...

class Bad(Task): ...      # TypeError: Bad must define ['run']

Choosing the right tool

You want to

Use

React to subclass creation: register it, validate it, set defaults

__init_subclass__

Modify or register one class

A class decorator (a function that takes and returns a class)

React when a descriptor is attached to a class

__set_name__

Control instance creation (singleton, pooling)

A metaclass __call__, or a plain factory function

Change the class namespace before the body runs, or alter the class creation itself

A metaclass (__prepare__, __new__)

Metaclasses you already use: abc.ABCMeta (abstract classes), enum.EnumMeta (enums), and the metaclasses behind many ORMs and web frameworks. Writing your own is rare; recognizing them when you read library code is the useful skill.

6. Dataclasses

The @dataclass decorator writes __init__, __repr__, and __eq__ for you from a list of annotated fields, so classes that mainly hold data need no boilerplate.

This decorator uses the dunder methods and this week's decorator idea: it takes a class, adds methods to it, and returns it.

from dataclasses import dataclass, asdict

@dataclass
class Point:
    x: float
    y: float = 0.0        # a field with a default

p = Point(1.5)
print(p)                      # Point(x=1.5, y=0.0)
print(p == Point(1.5, 0.0))   # True: __eq__ compares the fields as a tuple
print(asdict(p))              # {'x': 1.5, 'y': 0.0}

Fields without defaults must come before fields with defaults, as with function parameters.

Mutable defaults need default_factory

The mutable default trap applies here, and dataclass refuses it outright:

from dataclasses import field

@dataclass
class BadD:
    items: list = []
# ValueError: mutable default <class 'list'> for field items is not allowed: use default_factory

@dataclass
class Basket:
    items: list = field(default_factory=list)   # a fresh list per instance

a, b = Basket(), Basket()
a.items.append(1)
print(a, b)       # Basket(items=[1]) Basket(items=[])

Options on the decorator

Option

Effect

order=True

Adds <, <=, >, >=, comparing fields in order as tuples

frozen=True

Makes instances immutable and, with eq=True, hashable

slots=True

Uses __slots__: smaller instances, no __dict__ (Python 3.10+)

kw_only=True

Makes every field keyword-only in __init__ (Python 3.10+)

eq=False

Keeps identity-based equality instead of generating __eq__

@dataclass(order=True, frozen=True, slots=True)
class Version:
    major: int
    minor: int = 0
    tags: tuple = field(default=(), compare=False)   # left out of ==, <, and hash

v1, v2 = Version(1, 2), Version(1, 10)
print(v1 < v2)                       # True: compares (1, 2) with (1, 10)
print(sorted([v2, v1]))              # [Version(major=1, minor=2, ...), Version(major=1, minor=10, ...)]
print(hash(v1) == hash(Version(1, 2)))   # True: frozen + eq makes it hashable
print(hasattr(v1, "__dict__"))       # False: slots

v1.major = 5      # FrozenInstanceError: cannot assign to field 'major'

A regular (non-frozen) dataclass with eq=True sets __hash__ to None, so instances are unhashable; this is the __eq__-without-__hash__ rule:

hash(Point(1))    # TypeError: unhashable type: 'Point'

To "change" a frozen instance, create a modified copy with replace:

from dataclasses import replace
print(replace(v1, minor=9))    # Version(major=1, minor=9, tags=())

Computed fields: __post_init__, InitVar, init=False

__post_init__ runs right after the generated __init__, so use it for validation and derived values. An InitVar is passed to __init__ and __post_init__ but is not stored as a field.

from dataclasses import InitVar

@dataclass
class Rect:
    w: float
    h: float
    scale: InitVar[float] = 1       # input only, not stored
    area: float = field(init=False) # computed, not a constructor argument

    def __post_init__(self, scale):
        self.w *= scale
        self.h *= scale
        self.area = self.w * self.h

print(Rect(2, 3, scale=2))    # Rect(w=4, h=6, area=24)

Validation belongs in __post_init__ too:

@dataclass
class Emp:
    name: str
    def __post_init__(self):
        if not self.name:
            raise ValueError("name required")

Keyword-only fields and inheritance

@dataclass(kw_only=True)
class Cfg:
    host: str = "localhost"
    port: int = 8080

print(Cfg(port=9000))    # Cfg(host='localhost', port=9000)
Cfg("x")                 # TypeError: Cfg.__init__() takes 1 positional argument but 2 were given

@dataclass
class Base2:
    a: int

@dataclass
class Child(Base2):      # fields from the base come first
    b: int = 0

print(Child(1, 2))       # Child(a=1, b=2)

Inspect a dataclass with fields(cls) and asdict(obj); both are in the dataclasses module.

When to use a dataclass

  • Dataclass: a typed record you create in your own code, where you want defaults, ordering, or immutability
  • NamedTuple: an immutable record that must also behave as a tuple (unpacking, indexing)
  • Plain class: the object has behavior and invariants that matter more than its fields
  • pydantic model: data arrives from outside (JSON, user input) and must be validated and converted; dataclasses do not check types at runtime

7. Protocol and ABCs

Both tools describe an interface; an ABC enforces it by inheritance at runtime, while a Protocol describes it by shape and is checked by type checkers.

Abstract base classes (nominal typing)

An ABC lists methods that subclasses must implement. A class that inherits from the ABC but leaves an @abstractmethod undefined cannot be instantiated.

from abc import ABC, abstractmethod
import math

class Shape(ABC):
    @abstractmethod
    def area(self) -> float: ...

    def describe(self) -> str:                # concrete method shared by all shapes
        return f"{type(self).__name__} with area {self.area():.2f}"

class Circle(Shape):
    def __init__(self, r): self.r = r
    def area(self): return math.pi * self.r ** 2

class Broken(Shape): pass

Shape()      # TypeError: Can't instantiate abstract class Shape without an implementation for abstract method 'area'
Broken()     # TypeError: Can't instantiate abstract class Broken without an implementation for abstract method 'area'
print(Circle(1).describe())    # Circle with area 3.14

The error happens at instantiation, so a missing method is caught the first time anyone builds the object, not when the missing method is called deep inside a program.

Useful ABC features:

  • Abstract properties: stack @property on top of @abstractmethod
  • Mixin methods: concrete methods built on the abstract ones, like describe above (this is how collections.abc.Sequence gave you __iter__ and __contains__.
  • Virtual subclasses: Shape.register(Tri) makes isinstance(Tri(), Shape) true without inheritance
class Tri:
    def area(self): return 0.5

print(isinstance(Tri(), Shape))    # False
Shape.register(Tri)
print(isinstance(Tri(), Shape))    # True: registered, but Tri inherits nothing

Protocols (structural typing)

A Protocol says "anything with these methods qualifies", with no inheritance required. This is static duck typing: the class Resource below never mentions SupportsClose.

from typing import Protocol

class SupportsClose(Protocol):
    def close(self) -> None: ...

class Resource:
    def close(self) -> None:
        print("closed")

class Wrong:
    def shut(self) -> None: ...

def shutdown(item: SupportsClose) -> None:
    item.close()

shutdown(Resource())     # accepted: has close()
shutdown(Wrong())        # rejected by the type checker

Running mypy on that file reports:

error: Argument 1 to "shutdown" has incompatible type "Wrong"; expected "SupportsClose"  [arg-type]

Python itself does not stop shutdown(Wrong()); it fails at runtime with an AttributeError on item.close(). A protocol's value comes from the checker, run in your editor and CI.

Add @runtime_checkable if you also want isinstance to test for the methods at runtime:

from typing import runtime_checkable
import io

@runtime_checkable
class SupportsClose(Protocol):
    def close(self) -> None: ...

print(isinstance(io.StringIO(), SupportsClose))   # True
print(isinstance(42, SupportsClose))              # False
print(isinstance(Wrong(), SupportsClose))         # False

A runtime check only confirms that the method names exist; it does not check argument types or return types.

Choosing between them

Question

ABC

Protocol

Must implementers inherit from it?

Yes (or be registered)

No

Can it supply shared, concrete methods?

Yes

Not usefully; it is an interface only

Where is a violation caught?

At instantiation, at runtime

By the type checker, before running

Works with classes you do not control?

Only through register

Yes, automatically

Typical use

A framework base class you want people to extend

A parameter type that accepts anything shaped right

A good rule: use a Protocol to type the inputs of your functions, so callers can pass anything compatible, and use an ABC for a base class that carries shared behavior and needs enforced overrides.

Lab: Build a retry, caching, and timing decorator library

You will build decorkit.py, a small library of five tools that use every idea from this week, and prove it correct with 13 pytest tests that run in milliseconds. The code below passed all 13 tests on Python 3.13.

Specification

Tool

Kind

Behavior

Timer

Class using ContextDecorator

Works as a with block and as a decorator; exposes .elapsed

timed

Function decorator, bare or with arguments

Records calls, last_elapsed, total_elapsed on the function; optional report callback

retry

Decorator with arguments

Retries on chosen exceptions with attempts, delay, backoff, and an on_retry hook

ttl_cache

Decorator with arguments

Caches results for ttl seconds, evicts least recently used past maxsize, thread-safe, has cache_info() and cache_clear()

CallCounter

Class-based decorator

Counts calls; works on plain functions and on methods

Design decisions to notice as you read the code:

  • Time is injected. timed, Timer, and ttl_cache take a clock, and retry takes a sleep, so tests control time instead of waiting for it
  • Every wrapper uses functools.wraps, so names, docstrings, and __wrapped__ survive
  • State lives on the wrapper function (calls, cache_info). Because wraps copies a function's __dict__, these attributes stay visible when decorators are stacked
  • Failures are handled deliberately: timed records time in a finally, retry re-raises the last error, and ttl_cache never stores an exception

Step 1: Timer and timed

Create decorkit.py and start with the imports, Timer, and timed.

"""Week 2 lab: a small decorator library with timed, retry, ttl_cache and CallCounter."""
from __future__ import annotations

import functools
import threading
import time
import types
from collections import OrderedDict
from contextlib import ContextDecorator
from typing import Any, Callable, NamedTuple


class Timer(ContextDecorator):
    """Measure elapsed time; works as a `with` block and as a decorator."""

    def __init__(self, clock: Callable[[], float] = time.perf_counter):
        self._clock = clock
        self.elapsed = 0.0

    def __enter__(self) -> Timer:
        self._start = self._clock()
        return self

    def __exit__(self, exc_type, exc, tb) -> bool:
        self.elapsed = self._clock() - self._start
        return False


def timed(func=None, *, report: Callable[[str, float], None] | None = None,
          clock: Callable[[], float] = time.perf_counter):
    """Record call count and timing on the wrapper; optionally report every call.

    Works bare (`@timed`) and with arguments (`@timed(report=print)`).
    """
    def decorate(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            start = clock()
            try:
                return fn(*args, **kwargs)
            finally:
                elapsed = clock() - start
                wrapper.calls += 1
                wrapper.last_elapsed = elapsed
                wrapper.total_elapsed += elapsed
                if report is not None:
                    report(fn.__name__, elapsed)

        wrapper.calls = 0
        wrapper.last_elapsed = 0.0
        wrapper.total_elapsed = 0.0
        return wrapper

    return decorate if func is None else decorate(func)

Try it (timings will differ slightly on your machine):

>>> from decorkit import timed, Timer
>>> import time
>>> @timed(report=lambda name, s: print(f"{name}: {s:.3f}s"))
... def nap():
...     time.sleep(0.05)
>>> nap()
nap: 0.050s
>>> nap.calls, round(nap.total_elapsed, 2)
(1, 0.05)
>>> with Timer() as t:
...     sum(range(1_000_000))
>>> t.elapsed > 0
True

The pattern return decorate if func is None else decorate(func) is the optional-arguments trick from the decorators section: with @timed(...) the first argument is empty, so a decorator is returned; with bare @timed, the function arrives directly.

Step 2: retry

This is a decorator factory: three nested functions, where the outermost takes the settings, the middle takes the function, and the innermost is the wrapper that runs on every call. Append it to decorkit.py.

def retry(attempts: int = 3, *, delay: float = 0.0, backoff: float = 1.0,
          exceptions: type[BaseException] | tuple[type[BaseException], ...] = Exception,
          on_retry: Callable[[int, BaseException], None] | None = None,
          sleep: Callable[[float], None] = time.sleep):
    """Retry a function that raises one of `exceptions`, up to `attempts` calls in total.

    The wait starts at `delay` and is multiplied by `backoff` after each failure.
    The last exception is re-raised once the attempts are used up.
    """
    if attempts < 1:
        raise ValueError("attempts must be at least 1")

    def decorate(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            wait = delay
            for attempt in range(1, attempts + 1):
                try:
                    return fn(*args, **kwargs)
                except exceptions as exc:
                    if attempt == attempts:
                        raise
                    if on_retry is not None:
                        on_retry(attempt, exc)
                    if wait > 0:
                        sleep(wait)
                    wait *= backoff

        return wrapper

    return decorate

What to notice:

  • The attempts < 1 check runs when the decorator is applied, not when the function is called, so a bad setting fails at import time
  • A bare raise inside except re-raises the original exception with its traceback intact
  • With delay=1, backoff=2 the waits are 1 s, then 2 s, then 4 s, and so on
  • Only exceptions listed in exceptions are retried; anything else escapes on the first call
>>> from decorkit import retry
>>> import random
>>> @retry(attempts=4, delay=0.1, backoff=2, exceptions=ConnectionError,
...        on_retry=lambda n, e: print(f"attempt {n} failed: {e}"))
... def call_api():
...     if random.random() < 0.7:
...         raise ConnectionError("timeout")
...     return "ok"

Step 3: ttl_cache

This combines a closure (the cache lives in the enclosing scope), a lock, and an OrderedDict that doubles as a least-recently-used list. Append it to decorkit.py.

class CacheInfo(NamedTuple):
    hits: int
    misses: int
    maxsize: int
    currsize: int


def ttl_cache(ttl: float = 60.0, maxsize: int = 128,
              clock: Callable[[], float] = time.monotonic):
    """Cache results for `ttl` seconds, keeping at most `maxsize` entries (least recently used out).

    Arguments must be hashable. Exceptions are not cached. Thread-safe.
    """
    if ttl <= 0:
        raise ValueError("ttl must be positive")
    if maxsize < 1:
        raise ValueError("maxsize must be at least 1")

    def decorate(fn):
        store: OrderedDict[Any, tuple[float, Any]] = OrderedDict()
        lock = threading.Lock()
        stats = {"hits": 0, "misses": 0}

        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            key = (args, tuple(sorted(kwargs.items())))
            with lock:
                entry = store.get(key)
                if entry is not None and clock() - entry[0] < ttl:
                    store.move_to_end(key)
                    stats["hits"] += 1
                    return entry[1]
                stats["misses"] += 1
            value = fn(*args, **kwargs)  # computed outside the lock so slow calls don't block others
            with lock:
                store[key] = (clock(), value)
                store.move_to_end(key)
                while len(store) > maxsize:
                    store.popitem(last=False)
            return value

        def cache_info() -> CacheInfo:
            with lock:
                return CacheInfo(stats["hits"], stats["misses"], maxsize, len(store))

        def cache_clear() -> None:
            with lock:
                store.clear()
                stats["hits"] = stats["misses"] = 0

        wrapper.cache_info = cache_info
        wrapper.cache_clear = cache_clear
        return wrapper

    return decorate

How the pieces fit:

  • store, lock, and stats are free variables captured by wrapper, cache_info, and cache_clear: one private cache per decorated function (closures)
  • The key is built from positional arguments plus sorted keyword items, so add(1, b=2) and add(1, b=3) are different entries
  • move_to_end marks an entry as most recently used, and popitem(last=False) removes the oldest
  • The lock is released while fn runs, so a slow call does not block other threads; the trade-off is that two threads can compute the same missing value at once
  • cache_info is a NamedTuple, so it compares equal to a plain tuple, which the tests use
>>> from decorkit import ttl_cache
>>> @ttl_cache(ttl=5, maxsize=2)
... def square(x):
...     print("computing", x)
...     return x * x
>>> square(4); square(4)
computing 4
16
16
>>> square.cache_info()
CacheInfo(hits=1, misses=1, maxsize=2, currsize=1)

Step 4: CallCounter (class-based)

A class used as a decorator stores the function and its state in an instance and implements __call__. On its own it breaks on methods, because an instance is not a function and so is not a descriptor; adding __get__ fixes that, as covered in the descriptors section.

class CallCounter:
    """Class-based decorator that counts calls; also works on methods."""

    def __init__(self, func):
        functools.update_wrapper(self, func)
        self._func = func
        self.calls = 0

    def __call__(self, *args, **kwargs):
        self.calls += 1
        return self._func(*args, **kwargs)

    def __get__(self, instance, owner=None):
        if instance is None:
            return self
        return types.MethodType(self, instance)

functools.update_wrapper is what wraps calls internally; use it directly when the wrapper is an object rather than a function. types.MethodType(self, instance) builds a bound method that passes instance as the first argument. The counter is per decorated function, so calls from every instance of the class add up.

Step 5: Prove it with tests

Save this as test_decorkit.py next to decorkit.py, install the runner with pip install pytest, and run pytest -q. All 13 tests should pass.

The FakeClock is the key technique: it is a callable that returns a number you move forward by hand, so expiry, timing, and backoff are tested without sleeping.

import pytest
from decorkit import CallCounter, Timer, retry, timed, ttl_cache


class FakeClock:
    """A controllable clock so tests never sleep."""

    def __init__(self):
        self.now = 0.0

    def __call__(self):
        return self.now

    def advance(self, seconds):
        self.now += seconds


# --- Timer and timed ----------------------------------------------------------
def test_timer_as_context_manager_and_decorator():
    clock = FakeClock()
    with Timer(clock) as t:
        clock.advance(1.5)
    assert t.elapsed == 1.5

    timer = Timer(clock)

    @timer
    def work():
        clock.advance(2)

    work()
    assert timer.elapsed == 2


def test_timed_records_calls_and_elapsed():
    clock = FakeClock()

    @timed(clock=clock)
    def work():
        """Do work."""
        clock.advance(2.5)
        return "done"

    assert work() == "done"
    assert work.calls == 1 and work.last_elapsed == 2.5
    work()
    assert work.calls == 2 and work.total_elapsed == 5.0
    assert work.__name__ == "work" and work.__doc__ == "Do work."


def test_timed_works_bare_and_reports_even_on_failure():
    @timed
    def one():
        return 1

    assert one() == 1 and one.calls == 1

    reports = []

    @timed(report=lambda name, elapsed: reports.append(name), clock=FakeClock())
    def boom():
        raise RuntimeError("x")

    with pytest.raises(RuntimeError):
        boom()
    assert reports == ["boom"] and boom.calls == 1


# --- retry ----------------------------------------------------------------------
def test_retry_succeeds_after_failures_with_backoff():
    calls, sleeps = [], []

    @retry(attempts=3, delay=1, backoff=2, sleep=sleeps.append)
    def flaky():
        calls.append(1)
        if len(calls) < 3:
            raise ConnectionError("down")
        return "ok"

    assert flaky() == "ok"
    assert len(calls) == 3
    assert sleeps == [1, 2]


def test_retry_reraises_last_error_and_calls_hook():
    seen = []

    @retry(attempts=2, on_retry=lambda n, e: seen.append((n, str(e))), sleep=lambda s: None)
    def always_fails():
        raise ValueError("bad")

    with pytest.raises(ValueError, match="bad"):
        always_fails()
    assert seen == [(1, "bad")]


def test_retry_only_catches_listed_exceptions():
    calls = []

    @retry(attempts=3, exceptions=(KeyError,), sleep=lambda s: None)
    def wrong_kind():
        calls.append(1)
        raise TypeError("not retried")

    with pytest.raises(TypeError):
        wrong_kind()
    assert len(calls) == 1


def test_retry_validates_attempts():
    with pytest.raises(ValueError):
        retry(attempts=0)


# --- ttl_cache ------------------------------------------------------------------
def test_ttl_cache_hits_and_expiry():
    clock = FakeClock()
    calls = []

    @ttl_cache(ttl=10, clock=clock)
    def square(x):
        calls.append(x)
        return x * x

    assert square(3) == 9 and square(3) == 9
    assert calls == [3]
    assert square.cache_info() == (1, 1, 128, 1)
    clock.advance(11)
    assert square(3) == 9
    assert calls == [3, 3]


def test_ttl_cache_evicts_least_recently_used():
    calls = []

    @ttl_cache(ttl=100, maxsize=2, clock=FakeClock())
    def ident(x):
        calls.append(x)
        return x

    ident(1); ident(2); ident(1); ident(3)    # 2 is the least recently used: evicted
    ident(1)                                  # still cached
    ident(2)                                  # was evicted: computed again
    assert calls == [1, 2, 3, 2]


def test_ttl_cache_keys_include_keyword_arguments_and_clear():
    calls = []

    @ttl_cache(ttl=100, clock=FakeClock())
    def add(a, b=0):
        calls.append((a, b))
        return a + b

    add(1, b=2); add(1, b=2); add(1, b=3)
    assert calls == [(1, 2), (1, 3)]
    add.cache_clear()
    assert add.cache_info().currsize == 0


def test_ttl_cache_does_not_cache_exceptions():
    calls = []

    @ttl_cache(ttl=100, clock=FakeClock())
    def fragile():
        calls.append(1)
        raise OSError("nope")

    for _ in range(2):
        with pytest.raises(OSError):
            fragile()
    assert len(calls) == 2


# --- CallCounter ---------------------------------------------------------------
def test_call_counter_on_function_and_method():
    @CallCounter
    def add(a, b):
        return a + b

    assert add(1, 2) == 3 and add.calls == 1 and add.__name__ == "add"

    class Greeter:
        @CallCounter
        def hi(self, name):
            return f"hi {name}"

    g = Greeter()
    assert g.hi("bo") == "hi bo"
    assert Greeter.hi.calls == 1


# --- composing decorators -------------------------------------------------------
def test_decorators_compose():
    clock = FakeClock()
    calls = []

    @timed(clock=clock)
    @retry(attempts=2, sleep=lambda s: None)
    @ttl_cache(ttl=10, clock=clock)
    def fetch(key):
        calls.append(key)
        if len(calls) == 1:
            raise OSError("first call fails")
        return key.upper()

    assert fetch("a") == "A"          # first attempt fails, the retry succeeds
    assert calls == ["a", "a"]
    assert fetch("a") == "A"          # served from the cache
    assert calls == ["a", "a"]
    assert fetch.calls == 2           # counted by the outer timed wrapper
    assert fetch.cache_info().hits == 1   # attributes survive functools.wraps

The last test shows composition at work: ttl_cache sits closest to the function, so an exception is never cached and retry calls it again; timed is outermost, so it counts calls to the whole stack.

Stretch goals

  1. Make timed, retry, and ttl_cache work on async def functions, using inspect.iscoroutinefunction to pick an async def wrapper and asyncio.sleep for the delay
  2. Add jitter to retry (a random extra wait of up to a given fraction of the delay) and test it by injecting a fake random function
  3. Type the library so mypy --strict passes, using ParamSpec and TypeVar so decorated functions keep their signatures; the current untyped wrappers report errors under --strict
  4. Add @ttl_cache support for unhashable arguments by falling back to repr of the arguments, then write down the risks of that choice
  5. Add a circuit_breaker decorator that stops calling a failing function for a cool-down period after N consecutive failures, tested with FakeClock
  6. Publish the library with a pyproject.toml and install it into a fresh virtual environment, as a preview of Week 6

Exercises, pitfalls, and checklist

Predict each output before running it; a wrong prediction points at the part of this tutorial to reread.

Predict the output

import functools
from contextlib import contextmanager
from dataclasses import dataclass

# 1
def make():
    fs = []
    for i in range(3):
        fs.append(lambda i=i: i * 10)
    return fs
print([f() for f in make()])

# 2
def a(f):
    def w(): return "a(" + f() + ")"
    return w
def b(f):
    def w(): return "b(" + f() + ")"
    return w
@a
@b
def f(): return "f"
print(f())

# 3
class D:
    def __get__(self, o, t=None): return "desc"
class C:
    x = D()
c = C(); c.x = "inst"
print(c.x)

# 4
@dataclass
class P:
    x: int
print(P(1) == P(1), P(1) is P(1))

# 5
@contextmanager
def cm():
    print("in")
    yield
    print("out")
with cm():
    print("body")

# 6
def deco(func):
    def wrapper(*a, **k):
        return func(*a, **k)
    return wrapper
@deco
def hello(): pass
print(hello.__name__)

# 7
@functools.lru_cache
def g(x):
    print("run", x); return x
g(1); g(1); g(2)
print(g.cache_info().hits)

#

Output

Why

1

[0, 10, 20]

The default argument i=i captures each value when the lambda is created

2

a(b(f))

Decorators apply bottom-up (a(b(f))), so a's wrapper is the outermost layer

3

inst

D has no __set__, so it is a non-data descriptor and the instance value wins

4

True False

A dataclass generates __eq__ over its fields, but two instances are still two objects

5

in, body, out (one per line)

Code before yield is setup, the with body runs at the yield, the rest is cleanup

6

wrapper

Without functools.wraps, the decorated function takes the wrapper's name

7

run 1, run 2, then 1

The second g(1) is a cache hit, so run 1 prints only once

Hands-on exercises

  1. Closure versus class: write make_accumulator() that returns an add(x) function and a total() function sharing one running total through nonlocal. Then rewrite it as a class and list one advantage of each version.
  2. @memoize: write your own cache decorator that supports keyword arguments and exposes the cache dict as wrapper.cache. Compare its behavior to functools.lru_cache on fib(35).
  3. @deprecated("use new_func"): a decorator with an argument that issues a DeprecationWarning (with warnings.warn, stacklevel=2) on each call and keeps the function's metadata. Test it with pytest.warns.
  4. temp_env(**variables): set environment variables inside a with block and restore the originals afterward, including when the block raises. Write it twice, once as a class and once with @contextmanager.
  5. Typed descriptors: build Typed(kind) and Range(low, high) descriptors with __set_name__, and use them on a Product class with name, price, and stock attributes. Check that bad values raise TypeError or ValueError.
  6. Plugin registry: write a base class whose subclasses register under a name= class keyword through __init_subclass__, raising an error on duplicate names. Then redo it with a class decorator and compare.
  7. Dataclass model: model an Order holding a list of frozen LineItem dataclasses, with a computed total field set in __post_init__, and make orders sortable by total.
  8. Protocol check: define class Reader(Protocol): def read(self) -> str: ... and a function load(source: Reader) -> str. Run mypy to confirm it accepts an io.StringIO and rejects an int.

Common pitfalls

Pitfall

Fix

Forgetting functools.wraps

Add it to every wrapper; the name, docstring, and signature introspection depend on it

Writing @retry where @retry() is needed

Use the func=None pattern so both forms work, or fail loudly when a function arrives as the first argument

Late-binding closures in a loop

Capture with a default argument, functools.partial, or a factory function

Class-based decorator on a method fails with a missing self

Implement __get__ that returns types.MethodType(self, instance), or use a function decorator

yield without try/finally in @contextmanager

Wrap the yield so cleanup runs when the block raises

Storing per-instance values on a descriptor object

Store them on the instance (obj.__dict__ or a private attribute)

functools.lru_cache on a method

The cache keeps every self alive; cache a module-level function, or use a per-instance cache

Mutable default in a dataclass field

Use field(default_factory=...)

Reaching for a metaclass first

Try __init_subclass__, a class decorator, or __set_name__ before a metaclass

Expecting isinstance on a Protocol to check signatures

It only checks that the method names exist; use a type checker for the rest

Caching calls with unhashable arguments

Convert them to tuples, or key the cache on an identifier

Checklist

  • I can explain what a free variable is, and when nonlocal is required
  • I can write a decorator three ways: plain, with arguments, and class-based, each preserving metadata
  • I can predict the order in which stacked decorators wrap and run
  • I can write a context manager as a class and with @contextmanager, and explain what returning True from __exit__ does
  • I can explain the lookup order for data descriptors, instance attributes, and non-data descriptors
  • I can explain why self is passed automatically to a method
  • I can list the steps that run when a class statement executes
  • I can choose between a dataclass, a NamedTuple, a plain class, and a pydantic model
  • I can choose between an ABC and a Protocol for a given interface
  • My decorkit library passes all 13 tests, and I completed at least one stretch goal

Next week: iterators, generators, and functional patterns. Generators are built from the same ideas as this week's @contextmanager: a function that pauses at yield and resumes later.

Advanced Functions and Object-Oriented Python

Level: Intermediate to advanced Suggested time: 5–7 hours Python: 3.10+ (examples use standard-library features) This week moves beyond w...