1. Closures in Python
1.1 What is a Closure?
A closure is a function that remembers variables from its enclosing scope, even after the outer function has finished executing.
Closures are useful for:
Maintaining state without global variables.
Creating configurable functions.
Implementing decorators.
Building function factories.
1.2 Understanding Closures
Consider this example:
def outer_function(message):
def inner_function():
print(message)
return inner_function
my_function = outer_function("Hello Python!")
my_function()Output:
Hello Python!Explanation:
outer_function()receives a message.It defines an inner function.
The inner function uses the outer function's variable.
The outer function returns the inner function.
Even after
outer_function()finishes, the inner function remembersmessage.
That is a closure.
1.3 Closure with State
def counter():
count = 0
def increment():
nonlocal count
count += 1
return count
return increment
my_counter = counter()
print(my_counter())
print(my_counter())
print(my_counter())Output:
1
2
3Important concept: nonlocal allows an inner function to modify a variable defined in its enclosing function.
Without nonlocal, Python treats an assignment to count as a new local variable.
1.4 Practical Example: Bank Account
def create_account(initial_balance):
balance = initial_balance
def deposit(amount):
nonlocal balance
balance += amount
return balance
def withdraw(amount):
nonlocal balance
if amount <= balance:
balance -= amount
else:
print("Insufficient balance")
return balance
return deposit, withdraw
deposit, withdraw = create_account(1000)
print(deposit(500))
print(withdraw(200))Output:
1500
1300Interview question: What is the difference between a closure and a global variable?
Answer: A closure maintains access to variables in its enclosing lexical scope. A global variable belongs to the module-level namespace and can be accessed by functions throughout that module.
2. Decorators in Python
2.1 What is a Decorator?
A decorator is a function that modifies or extends the behavior of another function without changing its original source code.
Decorators are commonly used for:
Logging
Authentication
Authorization
Performance monitoring
Caching
Retry mechanisms
Input validation
2.2 Basic Decorator
def my_decorator(func):
def wrapper():
print("Before function execution")
func()
print("After function execution")
return wrapper
@my_decorator
def say_hello():
print("Hello World!")
say_hello()Output:
Before function execution
Hello World!
After function executionThe following:
@my_decorator
def say_hello():
print("Hello World!")is equivalent to:
def say_hello():
print("Hello World!")
say_hello = my_decorator(say_hello)2.3 Decorators with Arguments
A decorator should support functions that accept different parameters.
def my_decorator(func):
def wrapper(*args, **kwargs):
print("Function started")
result = func(*args, **kwargs)
print("Function completed")
return result
return wrapper
@my_decorator
def add(a, b):
return a + b
print(add(10, 20))Output:
Function started
Function completed
30Important:
*argsaccepts positional arguments.**kwargsaccepts keyword arguments.return resultpreserves the original function's return value.
2.4 Decorators with Their Own Arguments
Sometimes we want to configure the decorator itself.
Example: Repeat a function a specified number of times.
def repeat(times):
def decorator(func):
def wrapper(*args, **kwargs):
for _ in range(times):
result = func(*args, **kwargs)
return result
return wrapper
return decorator
@repeat(3)
def greet():
print("Hello!")
greet()Output:
Hello!
Hello!
Hello!Notice the three levels:
repeat(times)receives decorator configuration.decorator(func)receives the original function.wrapper(*args, **kwargs)executes the function.
2.5 Preserve Function Metadata Using functools.wraps
Decorators can overwrite the original function's name and documentation.
import functools
def my_decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
return func(*args, **kwargs)
return wrapper
@my_decorator
def calculate():
"""Calculate something."""
return 100
print(calculate.__name__)
print(calculate.__doc__)Output:
calculate
Calculate something.Always consider using functools.wraps when writing decorators.
2.6 Class-Based Decorators
Python also allows us to implement decorators using classes.
A class-based decorator implements the __call__() method.
import functools
class Logger:
def __init__(self, func):
functools.update_wrapper(self, func)
self.func = func
def __call__(self, *args, **kwargs):
print(f"Calling {self.func.__name__}")
result = self.func(*args, **kwargs)
print("Execution completed")
return result
@Logger
def add(a, b):
return a + b
print(add(10, 20))Output:
Calling add
Execution completed
30The class instance behaves like a function because it implements __call__().
2.7 Class-Based Decorator with Arguments
import functools
class Repeat:
def __init__(self, times):
self.times = times
def __call__(self, func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
result = None
for _ in range(self.times):
result = func(*args, **kwargs)
return result
return wrapper
@Repeat(3)
def greet():
print("Welcome!")
greet()This approach is useful when a decorator requires configuration or maintains internal state.
3. The functools Module
The functools module provides tools for working with higher-order functions and callable objects.
3.1 functools.wraps()
Preserves the metadata of the original function.
from functools import wrapsUse it when creating custom decorators.
3.2 functools.lru_cache()
Caches the result of function calls.
from functools import lru_cache
@lru_cache(maxsize=128)
def fibonacci(n):
if n < 2:
return n
return fibonacci(n - 1) + fibonacci(n - 2)
print(fibonacci(10))
print(fibonacci.cache_info())Output:
55
CacheInfo(hits=8, misses=11, maxsize=128, currsize=11)Benefits:
Reduces repeated calculations.
Improves performance.
Useful for recursive functions.
The arguments must be hashable, and cached results should be safe to reuse.
3.3 functools.cache()
An unbounded caching decorator.
from functools import cache
@cache
def square(n):
print("Calculating...")
return n * n
print(square(5))
print(square(5))The calculation runs only once for the same argument.
3.4 functools.partial()
Creates a new function by fixing some arguments of an existing function.
from functools import partial
def multiply(a, b):
return a * b
double = partial(multiply, 2)
print(double(10))
print(double(20))Output:
20
403.5 functools.singledispatch()
Allows a function to have different implementations based on the type of its first argument.
from functools import singledispatch
@singledispatch
def display(value):
print("Default:", value)
@display.register
def _(value: int):
print("Integer:", value)
@display.register
def _(value: list):
print("List:", value)
display(100)
display([1, 2, 3])
display("Python")This is useful when implementing type-specific behavior.
4. Context Managers and contextlib
4.1 What is a Context Manager?
A context manager manages resources and ensures that cleanup operations happen correctly.
Common examples:
Opening and closing files.
Managing database connections.
Managing locks.
Handling transactions.
Managing temporary resources.
Python provides the with statement for context managers.
4.2 File Handling Example
with open("sample.txt", "w") as file:
file.write("Learning Python")
The file is automatically closed when the with block exits, including when an exception occurs.
4.3 Creating a Custom Context Manager
A context manager class implements:
__enter__()__exit__()
class MyContext:
def __enter__(self):
print("Entering context")
return self
def __exit__(self, exc_type, exc_value, traceback):
print("Exiting context")
with MyContext():
print("Inside context")Output:
Entering context
Inside context
Exiting contextThe __exit__() method receives exception information.
Returning True from __exit__() suppresses the exception. Returning False or None allows it to propagate.
4.4 Using contextlib.contextmanager
The contextlib module provides a simpler way to create context managers using generators.
from contextlib import contextmanager
@contextmanager
def my_context():
print("Starting")
try:
yield
finally:
print("Cleaning up")
with my_context():
print("Executing task")Output:
Starting
Executing task
Cleaning upThe yield statement separates setup from the managed block and cleanup.
4.5 Other Useful contextlib Features
closing(): Ensures that an object with a close() method is closed.
from contextlib import closing
with closing(open("sample.txt")) as file:
print(file.read())suppress(): Suppresses specified exceptions.
from contextlib import suppress
with suppress(FileNotFoundError):
open("missing.txt")ExitStack: Manages multiple context managers dynamically.
from contextlib import ExitStack
with ExitStack() as stack:
files = [
stack.enter_context(open("a.txt")),
stack.enter_context(open("b.txt"))
]
print("Both files are open")5. Descriptors in Python
5.1 What is a Descriptor?
A descriptor is an object that defines one or more of these methods:
__get__()__set__()__delete__()
Descriptors control how attributes are accessed, assigned, and deleted.
They are used internally by properties, methods, and many Python frameworks.
5.2 Basic Descriptor Example
class Age:
def __get__(self, instance, owner):
return instance._age
def __set__(self, instance, value):
if value < 0:
raise ValueError("Age cannot be negative")
instance._age = value
class Person:
age = Age()
person = Person()
person.age = 25
print(person.age)Output:
25Trying to assign a negative age raises a ValueError.
5.3 Data and Non-Data Descriptors
A data descriptor implements __set__() or __delete__().
A non-data descriptor implements only __get__().
Data descriptors generally take precedence over an instance's dictionary when resolving attributes.
5.4 Practical Uses
Descriptors are useful for:
Attribute validation.
Type checking.
Lazy loading.
Reusable properties.
ORM field management.
Framework-level attribute management.
6. Metaclasses in Python
6.1 What is a Metaclass?
A metaclass is a class whose instances are classes.
In Python, the default metaclass is type.
class Employee:
pass
print(type(Employee))Output:
<class 'type'>Think of it this way:
Objects are created from classes.
Classes are created by metaclasses.
6.2 Creating a Custom Metaclass
class MyMeta(type):
def __new__(cls, name, bases, namespace):
print(f"Creating class: {name}")
return super().__new__(
cls, name, bases, namespace
)
class Employee(metaclass=MyMeta):
passOutput:
Creating class: EmployeeThe __new__() method participates in class creation.
6.3 Practical Example: Automatically Register Classes
class PluginMeta(type):
registry = {}
def __new__(cls, name, bases, namespace):
new_class = super().__new__(
cls, name, bases, namespace
)
if name != "Plugin":
cls.registry[name] = new_class
return new_class
class Plugin(metaclass=PluginMeta):
pass
class EmailPlugin(Plugin):
pass
class PaymentPlugin(Plugin):
pass
print(PluginMeta.registry)This pattern is useful for plugin architectures and framework registration.
Important: Metaclasses are powerful but should be used only when simpler alternatives are insufficient.
7. init_subclass()
__init_subclass__() is called whenever a new subclass is created.
It provides a simpler alternative to many metaclass use cases.
Example
class Plugin:
registry = []
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
Plugin.registry.append(cls)
class EmailPlugin(Plugin):
pass
class PaymentPlugin(Plugin):
pass
print(Plugin.registry)This automatically registers subclasses.
When should you use it?
Use __init_subclass__() when you want to:
Validate subclasses.
Register implementations.
Enforce class conventions.
Initialize subclass-level configuration.
Prefer it over a metaclass when you only need to customize subclass creation.
8. Dataclasses
8.1 What is a Dataclass?
A dataclass simplifies the creation of classes primarily used to store data.
The @dataclass decorator automatically generates methods such as:
__init__()__repr__()__eq__()
Example Without Dataclass
class Employee:
def __init__(self, name, salary):
self.name = name
self.salary = salarySame Example Using Dataclass
from dataclasses import dataclass
@dataclass
class Employee:
name: str
salary: float
emp = Employee("Rahul", 50000)
print(emp)Output:
Employee(name='Rahul', salary=50000)8.2 Default Values
from dataclasses import dataclass
@dataclass
class Product:
name: str
price: float
quantity: int = 1
product = Product("Laptop", 50000)
print(product)8.3 default_factory
Use default_factory for mutable default values.
from dataclasses import dataclass, field
@dataclass
class Student:
name: str
subjects: list = field(default_factory=list)
s1 = Student("Amit")
s2 = Student("Neha")
s1.subjects.append("Python")
print(s1.subjects)
print(s2.subjects)Output:
['Python']
[]Each object receives its own list.
8.4 Frozen Dataclasses
from dataclasses import dataclass
@dataclass(frozen=True)
class Configuration:
host: str
port: int
config = Configuration("localhost", 8000)A frozen dataclass prevents normal attribute reassignment after initialization.
9. Protocols and Abstract Base Classes
These concepts help build flexible and maintainable application architectures.
9.1 Abstract Base Classes (ABCs)
An ABC defines a contract that subclasses must implement.
from abc import ABC, abstractmethod
class PaymentProcessor(ABC):
@abstractmethod
def process_payment(self, amount):
pass
class CreditCardPayment(PaymentProcessor):
def process_payment(self, amount):
print(f"Processing payment: {amount}")
payment = CreditCardPayment()
payment.process_payment(1000)A subclass that does not implement all abstract methods cannot be instantiated.
9.2 Why Use ABC?
Use ABC when:
You want explicit inheritance.
You want to enforce implementation requirements.
You need shared behavior.
You are designing a framework or application architecture.
9.3 Protocol
A Protocol supports structural subtyping.
This means a class can satisfy an interface by implementing the required methods, without explicitly inheriting from that interface.
from typing import Protocol
class PaymentProcessor(Protocol):
def process_payment(self, amount: float) -> None:
...
class CreditCardPayment:
def process_payment(self, amount: float) -> None:
print(f"Paid {amount}")
def checkout(
processor: PaymentProcessor,
amount: float
):
processor.process_payment(amount)
checkout(CreditCardPayment(), 500)The CreditCardPayment class does not need to inherit from PaymentProcessor.
9.4 ABC vs Protocol
| Feature | ABC | Protocol |
|---|---|---|
| Typing approach | Nominal | Structural |
| Explicit inheritance | Usually required | Not required |
| Abstract methods | Yes | Method signatures |
| Runtime enforcement | Yes, for abstract methods | Not by default |
| Main purpose | Enforce contracts | Flexible interfaces |
Use ABC when inheritance and runtime enforcement are important.
Use Protocol when you want flexible interfaces and better static type checking.
10. LAB PROJECT: Build a Retry, Caching and Timing Decorator Library
Project Objective
Build a reusable Python library that provides three decorators:
@retry– Automatically retries failed function calls.@cache_result– Caches function results.@timing– Measures function execution time.
These decorators should work with ordinary functions and support function arguments.
Project Structure
week2_project/
│
├── decorators.py
├── main.py
└── requirements.txtWe will use Python's standard library, so no external packages are required.
Step 1: Implement Timing Decorator
Create decorators.py.
import functools
import time
def timing(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
start = time.perf_counter()
try:
return func(*args, **kwargs)
finally:
elapsed = time.perf_counter() - start
print(
f"{func.__name__} executed in "
f"{elapsed:.4f} seconds"
)
return wrapperExplanation:
time.perf_counter()measures elapsed time.functools.wraps()preserves function metadata.finallyensures timing is printed even if the function raises an exception.
Step 2: Implement Retry Decorator
Add the following code to decorators.py.
def retry(max_attempts=3, delay=1, exceptions=(Exception,)):
def decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
for attempt in range(1, max_attempts + 1):
try:
return func(*args, **kwargs)
except exceptions:
if attempt == max_attempts:
raise
print(
f"Attempt {attempt} failed. "
f"Retrying..."
)
time.sleep(delay)
return wrapper
return decoratorUsage:
@retry(max_attempts=3, delay=2)
def connect_api():
print("Connecting to API")
raise ConnectionError("Server unavailable")
connect_api()The function will execute three times before the final exception is raised.
Important production considerations:
Retry only transient failures.
Use exponential backoff for real services.
Avoid retrying non-idempotent operations unless they are designed to be safe to retry.
Consider adding jitter and maximum delay.
Step 3: Implement Caching Decorator
Add this code to decorators.py.
def cache_result(maxsize=128):
def decorator(func):
cached_func = functools.lru_cache(
maxsize=maxsize
)(func)
@functools.wraps(func)
def wrapper(*args, **kwargs):
return cached_func(*args, **kwargs)
wrapper.cache_info = cached_func.cache_info
wrapper.cache_clear = cached_func.cache_clear
return wrapper
return decoratorUsage:
@cache_result(maxsize=100)
def calculate_square(number):
print("Calculating...")
return number * number
print(calculate_square(10))
print(calculate_square(10))
print(calculate_square.cache_info())The second call reuses the cached result.
Note: The cache uses function arguments as keys. Arguments must be hashable. Cached functions should generally avoid side effects, and cache invalidation should be considered when underlying data changes.
Step 4: Use All Three Decorators Together
Create main.py.
from decorators import timing, retry, cache_result
@timing
@retry(max_attempts=3, delay=1)
@cache_result(maxsize=100)
def fetch_data(user_id):
print(f"Fetching data for user {user_id}")
return {
"user_id": user_id,
"name": "Rahul"
}
print(fetch_data(101))
print(fetch_data(101))The decorators are applied from bottom to top:
cache_resultwraps the original function.retrywraps the cached function.timingwraps the retry-enabled function.
The execution flow is important because decorator order changes behavior.
Step 5: Test Your Library
Test the following scenarios:
A function that succeeds on the first attempt.
A function that fails twice and succeeds on the third attempt.
A function that always fails.
A function that returns the same result for repeated arguments.
A function that receives keyword arguments.
A function that takes several seconds to execute.
Two different functions using the same decorators.
Suggested testing framework: pytest.
11. Additional Advanced Challenges
Once the main project is complete, try these extensions:
Challenge 1: Exponential Backoff
Instead of a fixed delay, implement:
delay = initial_delay * (2 ** (attempt - 1))Challenge 2: Cache Expiration
Add a TTL (Time To Live) option so cached results expire after a specified number of seconds.
Challenge 3: Class-Based Timing Decorator
Implement timing using a class with __init__() and __call__().
Challenge 4: Logging Decorator
Create a decorator that logs:
Function name
Input arguments
Execution time
Return value
Exceptions
Avoid logging passwords, tokens, or other sensitive arguments.
Challenge 5: Async Retry Decorator
Extend the retry decorator to support async def functions using asyncio.sleep().
12. Week 2 Interview Questions
What is a closure in Python?
What is the purpose of
nonlocal?What is a decorator?
What is the difference between a decorator and a closure?
What is the purpose of
functools.wraps()?How do decorators with arguments work?
How do class-based decorators work?
What is the difference between
cacheandlru_cache?What is
functools.partial()?What is a context manager?
What are
__enter__()and__exit__()?What is the purpose of
contextlib?What is a descriptor?
What is the difference between data and non-data descriptors?
What is a metaclass?
What is the default metaclass in Python?
What is the purpose of
__init_subclass__()?What is a dataclass?
What is the purpose of
default_factory?What is a frozen dataclass?
What is an Abstract Base Class?
What is a Protocol?
What is the difference between ABC and Protocol?
What happens when multiple decorators are applied to a function?
How would you implement a retry decorator for an API?
13. Week 2 Completion Checklist
Understand closures and lexical scoping.
Implement decorators with and without arguments.
Implement class-based decorators.
Use
functools.wraps,lru_cache,cache, andpartial.Create custom context managers.
Understand descriptors.
Understand metaclasses and
__init_subclass__.Implement dataclasses.
- Understand ABCs and Protocols.
Complete the retry decorator.
Complete the caching decorator.
Complete the timing decorator.
Test the library using pytest.
Final Outcome
At the end of Week 2, you should be able to design reusable Python components and understand many of the advanced mechanisms used internally by professional Python frameworks and libraries.
Recommended hands-on study sequence
Day |
Topic |
Practical assignment |
|---|---|---|
Day 1 | Closures | Build a counter and bank account |
Day 2 | Decorators | Logging and authentication decorators |
Day 3 | functools | Fibonacci caching and partial functions |
Day 4 | Context managers | Database connection manager |
Day 5 | Descriptors and metaclasses | Attribute validation and plugin registry |
Day 6 | Dataclasses, ABC, Protocol | Payment processing architecture |
Day 7 | Lab project | Build and test the decorator library |
No comments:
Post a Comment