Closures and Decorators

Learn Python closures and decorators step by step: functions as values, nonlocal, functools.wraps, decorators with arguments and lru_cache.

• Intermediate
• 18 min read
• 4 objectives

Before this lessonLesson 8: Comprehensions and Iteration

You have probably already used a decorator without knowing how it works: the @property or @staticmethod line above a method, or @app.get("/") in a web framework. A decorator is a way to wrap extra behaviour (logging, timing, caching, access checks) around a function without editing the function itself.

Decorators look like magic until you see the two ideas underneath them: functions are ordinary values, and inner functions can remember variables from the function that created them (a closure). This lesson builds up from those two ideas, so by the end the @ syntax is just a shortcut you understand.

Functions are values

In Python a function is an object like a number or a list. You can store it in a variable, put it in a dict, pass it to another function, or return it. Notice the difference between shout (the function itself) and shout("hi") (calling it).

def shout(text):
    return text.upper() + "!"

def whisper(text):
    return text.lower() + "..."

speak = shout            # no parentheses: we copy the function, not its result
print(speak("hello"))

def greet(style, name):
    return style(f"hi {name}")

print(greet(shout, "Ada"))
print(greet(whisper, "Ada"))

handlers = {"loud": shout, "quiet": whisper}
print(handlers["quiet"]("STACKCONE"))

HELLO!
HI ADA!
hi ada...
stackcone...

Closures: functions that remember

A function defined inside another function can read the outer function's variables. The surprising part is that it keeps access to them after the outer function has returned. That combination of a function plus the variables it captured is called a closure. Here make_multiplier returns a new function each time, and each one remembers its own factor.

def make_multiplier(factor):
    def multiply(n):
        return n * factor     # factor comes from the enclosing call
    return multiply

double = make_multiplier(2)
triple = make_multiplier(3)
print(double(10), triple(10))
print(double.__closure__[0].cell_contents)

20 30
2

Closures are a lightweight alternative to a class when you only need one method and a little bit of state.

Changing captured state with nonlocal

Reading a captured variable is automatic, but assigning to it is not: Python would treat the name as a brand-new local variable. The nonlocal keyword tells Python you mean the variable from the enclosing function.

def make_counter():
    count = 0
    def increment():
        nonlocal count
        count += 1
        return count
    return increment

next_id = make_counter()
print(next_id(), next_id(), next_id())

other = make_counter()     # a fresh, independent count
print(other())

1 2 3
1

Your first decorator

A decorator is simply a function that takes a function and returns a new function (usually a closure that calls the original). The @name line above a def is shorthand for func = name(func).

def log_calls(func):
    def wrapper(*args, **kwargs):
        print(f"calling {func.__name__} with {args} {kwargs}")
        result = func(*args, **kwargs)
        print(f"{func.__name__} returned {result}")
        return result
    return wrapper

@log_calls
def add(a, b):
    return a + b

@log_calls
def greet(name, punctuation="!"):
    return f"Hello, {name}{punctuation}"

add(2, 3)
greet("Ada", punctuation="?")

calling add with (2, 3) {}
add returned 5
calling greet with ('Ada',) {'punctuation': '?'}
greet returned Hello, Ada?

The *args, **kwargs pair lets the wrapper accept any arguments and forward them unchanged, so one decorator works on any function. Always return result from the wrapper, or the decorated function will silently return None.

Keep the function's identity with functools.wraps

There is a subtle problem with the wrapper above: the decorated function now is wrapper, so its name and docstring are lost. That confuses debuggers, logs and documentation tools. functools.wraps copies the original metadata onto the wrapper. Use it in every decorator you write.

import functools

def plain(func):
    def wrapper(*args, **kwargs):
        return func(*args, **kwargs)
    return wrapper

def polite(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        return func(*args, **kwargs)
    return wrapper

@plain
def total(items):
    """Sum a list of prices."""
    return sum(items)

@polite
def average(items):
    """Average a list of prices."""
    return sum(items) / len(items)

print(total.__name__, total.__doc__)
print(average.__name__, average.__doc__)
print(average.__wrapped__([2, 4]))

wrapper None
average Average a list of prices.
3.0

A practical decorator: timing

Timing is a classic use case: you want to measure how long functions take without sprinkling timer code through each one. time.perf_counter() is the right clock for measuring short durations. Here we only print whether the call was slow, so the output is repeatable.

import functools
import time

def timed(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        try:
            return func(*args, **kwargs)
        finally:
            elapsed = time.perf_counter() - start
            label = "slow" if elapsed > 0.5 else "fast"
            print(f"{func.__name__} was {label}")
    return wrapper

@timed
def build_report(n):
    return sum(i * i for i in range(n))

print(build_report(10_000))

build_report was fast
333283335000

The try/finally means the timing message prints even if the function raises an exception.

Decorators that take arguments

What about @retry(times=3)? Because retry(times=3) is called first, it must return a decorator. That means three levels of functions: the outer one takes the settings, the middle one takes the function, and the inner one runs on each call.

import functools

def retry(times):
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(1, times + 1):
                try:
                    return func(*args, **kwargs)
                except ConnectionError as err:
                    print(f"attempt {attempt} failed: {err}")
            raise ConnectionError(f"gave up after {times} attempts")
        return wrapper
    return decorator

calls = {"n": 0}

@retry(times=3)
def fetch_orders():
    calls["n"] += 1
    if calls["n"] < 3:
        raise ConnectionError("timeout")
    return ["order-1", "order-2"]

print(fetch_orders())

attempt 1 failed: timeout
attempt 2 failed: timeout
['order-1', 'order-2']

Built-in decorators worth knowing

The standard library ships several ready-made decorators. functools.lru_cache (or the unbounded functools.cache) remembers results for arguments it has seen, which can turn an exponential recursive function into an instant one. You have also met @property, @classmethod and @staticmethod in the classes lessons, and @dataclass appears later in this course.

import functools

@functools.lru_cache(maxsize=None)
def fib(n):
    return n if n < 2 else fib(n - 1) + fib(n - 2)

print(fib(80))
print(fib.cache_info())

23416728348467685
CacheInfo(hits=78, misses=81, maxsize=None, currsize=81)

You can stack decorators. They apply bottom-up: @a above @b above def f means f = a(b(f)).

Recap

• Functions are values: you can pass them, return them and store them.
• A closure is an inner function that remembers variables from its enclosing scope; use nonlocal to reassign them.
• A decorator takes a function and returns a wrapped one; @deco means f = deco(f).
• Always use *args, **kwargs, return the result and apply @functools.wraps.
• A decorator with arguments is a function that returns a decorator; lru_cache is a ready-made one for memoisation.

# Write your solution here