my_list = [100, 200]
a, b = my_list
print(a)
print(b)100
200
More than you wanted to know about functions
Karsten Naert
November 15, 2025
You are already familiar with this syntax:
This is a way of unpacking the list into two variables. We could do this, because we know the list has precisely two elements.
Sometimes you don’t know how many variables to expect and you just want to grab whatever remains and put it in a list. You can do this with *name_of_list. An example:
first=1
second=2
tail=[3, 4]
--------------------------------------------------------------------------- ValueError Traceback (most recent call last) Cell In[5], line 1 ----> 1 first, second = [1] 2 print(f'{first=}') 3 print(f'{second=}') ValueError: not enough values to unpack (expected 2, got 1)
A neat trick is to unpack a list with one element:
Since the star grabs all the rest, it doesn’t make sense to do it twice:
Cell In[7], line 1 head, *body, *tail = [1, 2, 3, 4, 5] ^ SyntaxError: multiple starred expressions in assignment
Try to predict what each of these unpacking operations will produce, then test your predictions:
Also try unpacking with nested structures:
More info: read PEP 3132
A function is a piece of code that gets called by another piece of code. That means there are two parties involved:
Often you will act as both author and user of a function, but for functions from libraries or builtin functions, these may be different people.
Calling functions is actually quite a bit simpler than defining them, at least in full generality, so let’s start there.
You already know two ways to pass arguments:
f(1, 2, 3). We simply type the variable.f(a=1, b=2, c=3). Here the syntax is name=variable.The one rule here is that keyword arguments must always come after the positional arguments.
We will now see two extra ways to pass arguments: as args and kwargs.
The *args are a way to pass a bunch of arguments directly from a function into a list. Let’s say our function looks like this:
And let’s say we have a list with precisely three elements
We could call the function for instance like this
But we can also pass the list directly into the function with the unpacking operator:
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[12], line 2 1 the_list = [1, 2, 3, 4] ----> 2 f(*the_list) TypeError: f() takes 3 positional arguments but 4 were given
It’s perfectly fine to combine this with other arguments, and you can also use it multiple times:
A useful way to think about it is that * removes the brackets: g(*[1, 10], *[1, 10]) becomes g(1, 10, 1, 10).
For passing multiple keyword arguments at the same time, you can use the syntax **kwargs in the same way.
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[18], line 2 1 the_dict = {'b': 2, 'c': 3, 'd': 4} ----> 2 f(a=1, **the_dict) TypeError: f() got an unexpected keyword argument 'd'
This approach is often used to prepare the arguments in a dict where they can be easily manipulated and then pass them to the function in one go.
import requests
# Build the argument dictionary
kwargs = {
"url": "https://api.github.com/repos/psf/requests",
"headers": {"Accept": "application/vnd.github.v3+json"},
"params": {"per_page": 5}
}
# Call requests.get with the keyword arguments
response = requests.get(**kwargs)
print(response.status_code)
print(response.json())Practice using *args and **kwargs by completing these tasks:
flexible_sum(*args, **kwargs) that:
debug_call(func, *args, **kwargs) that:
Try it with: debug_call(flexible_sum, 1, 2, bonus=3)
Defining functions is actually much harder than using them, because you have to foresee all the ways in which a user will call your function and decide what to do in each case.
The functions structure for how it accepts its arguments (and what type it returns) is often called the signature of the function. By the end of this lecture we will see in full generality how complicated a function signature can get in Python1.
The arguments are often called parameters in the context of defining a function2. This is because in the function body, they should be considered blanks that are to be filled in by the arguments when the function is called, sometimes combined with default values.
This function has three parameters. Each of these can be passed as a position argument and as a keyword argument.
However, two of these parameters have a default value, which implies they can be omitted. Let’s look at some examples.
First, we can pass exactly one positional argument. Note that this is ok, because the other parameters have defaults.
Next, we will pass two arguments, both positional. These get grabbed to fill in the parameters x and y, while for the parameter z we fall back to its default value:
We can also call this function with purely keyword arguments. In this case the parameters can be provided based on their name:
We can also mix positional and keyword arguments:
It is not allowed to define a parameters twice, typically once with a positional argument and once more with a keyword argument.
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[25], line 1 ----> 1 f(2, x=1) TypeError: f() got multiple values for argument 'x'
Note that the function caller must always provide the positional arguments first and keyword arguments second, so the following syntax is illegal, regardless of how we define the function:
If not all parameters could be found, whether from positional, keyword arguments, or from their default values, we also get an error:
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[27], line 1 ----> 1 f() TypeError: f() missing 1 required positional argument: 'x'
You can use the following function to investigate the signature of functions. Note that each parameter is indeed POSITIONAL_OR_KEYWORD, some with and some without default.
import inspect
def reveal_signature(func):
sig = inspect.signature(func)
for name, param in sig.parameters.items():
if param.default is inspect._empty:
print(name, param.kind, '(No default)')
else:
print(name, param.kind, 'default =', param.default)
reveal_signature(f)x POSITIONAL_OR_KEYWORD (No default)
y POSITIONAL_OR_KEYWORD default = 10
z POSITIONAL_OR_KEYWORD default = 20
Use the reveal_signature function on some functions from third party libraries you know. See if you can find any new sorts of parameters!
The syntax x=5 means something different from the perspective of the caller versus the callee:
x with value 5;5 for this parameter, you are not declaring that it must be called as keyword argument.You can also define a function that will accept any number of positional arguments by using the unpacking operator. The convention is to use the name args.
When the function gets called, args will become a tuple with all the remaining positional arguments that could not be matched.
def f(x, *args):
print('x =', x, 'args =', args)
reveal_signature(f)
print()
f(1, 2, 3)
f(1, 2)
f(1)x POSITIONAL_OR_KEYWORD (No default)
args VAR_POSITIONAL (No default)
x = 1 args = (2, 3)
x = 1 args = (2,)
x = 1 args = ()
The unpacking operator does not accept arguments passed as keywords, so this would be illegal:
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[30], line 1 ----> 1 f(1, y=2) TypeError: f() got an unexpected keyword argument 'y'
This also doesn’t change the fact that all parameters must eventually receive a value:
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[31], line 1 ----> 1 f() TypeError: f() missing 1 required positional argument: 'x'
It is possible to combine this with parameters which have a default value, for instance:
def f(x, y=10, *args): # <-- weird! don't do this
print('x =', x, 'args =', args, 'y =', y)
reveal_signature(f)
print()
f(1, 2, 3, 4)
f(1, y=5)
f(1)x POSITIONAL_OR_KEYWORD (No default)
y POSITIONAL_OR_KEYWORD default = 10
args VAR_POSITIONAL (No default)
x = 1 args = (3, 4) y = 2
x = 1 args = () y = 5
x = 1 args = () y = 10
The previous example is legal but a little bit weird. You can provide y=5 or extra positional arguments 2, 3, 4 but there is no ways to provide both at the same time, because the obvious syntax f(1, 2, 3, 4, y=10) actually defines the argument y two times, once positional and once with keyword!
What if we did it the other way?
def f(x, *args, y=10):
print('x =', x, 'args =', args, 'y =', y)
reveal_signature(f)
print()
f(1, 2, 3, 4)
f(1, y=5)
f(1)x POSITIONAL_OR_KEYWORD (No default)
args VAR_POSITIONAL (No default)
y KEYWORD_ONLY default = 10
x = 1 args = (2, 3, 4) y = 10
x = 1 args = () y = 5
x = 1 args = () y = 10
Now something interesting has occurred: no matter how many positional arguments we provide to our function f, we will never be able to provide the parameter y as a positional argument, because *args will eat them all! That makes this a KEYWORD_ONLY parameter.
In the previous example our KEYWORD_ONLY parameter y has a default value, but this is absolutely not required!
def f(x, *args, y):
print('x =', x, 'args =', args, 'y =', y)
reveal_signature(f)
print()
f(1, 2, 3, 4, y=10)
f(1, y=5)
f(1, y=10)x POSITIONAL_OR_KEYWORD (No default)
args VAR_POSITIONAL (No default)
y KEYWORD_ONLY (No default)
x = 1 args = (2, 3, 4) y = 10
x = 1 args = () y = 5
x = 1 args = () y = 10
If there is no default value, and you don’t provide one, you get an error of course:
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[35], line 1 ----> 1 f(1, 2, 3) TypeError: f() missing 1 required keyword-only argument: 'y'
In principle you can give the argument with defaults and without defaults in any order, but it is a little strange to do it, I recommend combining all the parameters with defaults.
def f(x, *, a=10, b, c=12, d): # don't do this, prefer: b, d, a=10, c=12
print(f'{x=}, {a=}, {b=}, {c=}, {d=}')
reveal_signature(f)
f(1, a=1, b=2, c=3, d=4)x POSITIONAL_OR_KEYWORD (No default)
a KEYWORD_ONLY default = 10
b KEYWORD_ONLY (No default)
c KEYWORD_ONLY default = 12
d KEYWORD_ONLY (No default)
x=1, a=1, b=2, c=3, d=4
If you want to specify that the parameters are keyword-only but there should not be an unlimited number of positional arguments allowed, you can use this syntax:
x POSITIONAL_OR_KEYWORD (No default)
z KEYWORD_ONLY (No default)
u KEYWORD_ONLY default = 1
x = 1 z = 1 u = 3
The * plays the same role as *args but it won’t eat extra positional arguments.
Making all parameters keyword-only is a useful trick! This way you can always change your mind later about the order in which the parameters should appear without breaking existing code.
x KEYWORD_ONLY (No default)
y KEYWORD_ONLY (No default)
x = 1 y = 2
x = 1 y = 2
By analogy with *args there also exists **kwargs that will eat up all the keyword arguments that could not be matched with any other parameter.
def f(y=10, **kwargs):
print('y =', y, 'kwargs =', kwargs)
reveal_signature(f)
print()
f(3, z=10)
f(y=3, z=10)
f(y=3, z=10, u=123, v=1234)
f(z=10, y=3)
f(3)
f(z=10)y POSITIONAL_OR_KEYWORD default = 10
kwargs VAR_KEYWORD (No default)
y = 3 kwargs = {'z': 10}
y = 3 kwargs = {'z': 10}
y = 3 kwargs = {'z': 10, 'u': 123, 'v': 1234}
y = 3 kwargs = {'z': 10}
y = 3 kwargs = {}
y = 10 kwargs = {'z': 10}
We can combine this with all the rest:
def f(x, u=3, *args, z, y=10, **kwargs):
print('x =', x, 'u=', u, 'args =', args, 'y =', y, 'z =', z, 'kwargs =', kwargs)
reveal_signature(f)x POSITIONAL_OR_KEYWORD (No default)
u POSITIONAL_OR_KEYWORD default = 3
args VAR_POSITIONAL (No default)
z KEYWORD_ONLY (No default)
y KEYWORD_ONLY default = 10
kwargs VAR_KEYWORD (No default)
Finally, since Python 3.8 we have another syntax that can make arguments positional-only.
x POSITIONAL_ONLY (No default)
y POSITIONAL_ONLY (No default)
z POSITIONAL_OR_KEYWORD (No default)
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[45], line 1 ----> 1 f(1, y=2, z=3) TypeError: f() got some positional-only arguments passed as keyword arguments: 'y'
This makes this about the most general function signature:
def f(a, /, x, u=3, *args, z, y=10, **kwargs):
print('a =', a, 'x =', x, 'u=', u, 'args =', args, 'y =', y, 'z =', z, 'kwargs =', kwargs)
reveal_signature(f)a POSITIONAL_ONLY (No default)
x POSITIONAL_OR_KEYWORD (No default)
u POSITIONAL_OR_KEYWORD default = 3
args VAR_POSITIONAL (No default)
z KEYWORD_ONLY (No default)
y KEYWORD_ONLY default = 10
kwargs VAR_KEYWORD (No default)
Practice with different parameter types by creating these functions and testing them:
create_config(name, /, version=1.0, *, debug=False, **options) that:
name parameterversion parameter with default 1.0debug parameter with default False**optionslog_message(level, /, message, *details, timestamp=None, **metadata) that:
level parametermessage parameter*detailstimestamp parameter**metadataTry invalid calls to understand the error messages!
The role of the default values of the parameters can sometimes lead to confusion. Consider the following example:
The first time we call f, it starts from and empty list and appends 1. But the second time, the 1 is already appended and another 1 is appended.
This is because the default value is defined only once, when the function is defined. It is not defined every time the function is called. That means if it gets mutated, all subsequent function calls will see the mutated value. This can be exploited
def addition(x, y, _count=[0]): # don't do this
_count[0] += 1
print(f"Addition was called {_count[0]} times.")
return x + y
addition(1, 2)
addition(3, 4)
addition(5, 6)Addition was called 1 times.
Addition was called 2 times.
Addition was called 3 times.
11
… but there are better ways to achieve the same, with a class or a global variable if you have to.
So what if you actually want an empty list as default argument? Best practice is that you pass None and check for it inside the function:
This way you are guaranteed to start from a fresh list every time.
So the lesson to remember is this: default values should not be mutable.
Test your understanding of mutable default arguments:
Fix the Function: Rewrite the function above to avoid the mutable default argument problem.
Advanced Challenge: Create a function create_counter(start=0) that returns a function which increments and returns a counter. Each call to create_counter() should create an independent counter:
If you read a bit in the official Python documentation about the execution model you will learn that Python code is built out of blocks. There are many types of blocks but the most important ones are:
If you define a variable, it comes with a scope attached, this is the piece of code where the variable is visible from. The scope is usually just the same block, but for function blocks it is also the blocks contained in it.
To look up a variable, Python simply looks at the surrounding scope. In the next example we look up a variable y. The function f looks it up from it surrounding block:
When defining the function f, it registers that it needs to return the global variable y.
If we change the value of the variable after the function is defined, we will receive the adapted value. The function f() looks this value up in its higher scope.
We do not capture the value of y when the function is defined. That means if y changes value before the function is called, f will still look up its latest value.
The situation is very different when we assign a value to y inside the function3. In the next example another, second variable y is created, which only exists inside the inner scope. So the example has two variable, with the same name but a different scope:
The command return y gives us the value of the local variable y and ignores the global y.
So how does Python decide whether to return the local or global variable? The rule is: if the variable is assigned, or technically, if the name is bound, anywhere in a block, that variable is considered local.
Consider this example:
--------------------------------------------------------------------------- UnboundLocalError Traceback (most recent call last) Cell In[53], line 8 5 return y 7 y = 20 ----> 8 f(), y Cell In[53], line 5, in f() 3 if False: 4 y = 1 ----> 5 return y UnboundLocalError: cannot access local variable 'y' where it is not associated with a value
Because y is defined inside the block f, the variable is considered local. It does not matter that we hit this line of code or not! So when we return y, Python tries to return a local variable that has not received a value yet. Technically, the name of the local variable has not been bound to a value. The result is an ugly UnboundLocalError.
If we don’t want Python to consider a variable as local, the command global y will change this. Here, the same thing is true: if Python finds this anywhere in the function, it will treat all occurrences of y as global. This can overrule the earlier rules, which allows you to change a global variable from within a function after all.
What matters is that you declare y to be global. It doesn’t matter that you hit this line or code or not.
A function can return another function:
The variable N is a free variable inside the function add. The function can then look up its value.
Here is some helper code to understand how this works:
def reveal(func):
print('Free variables: ', func.__code__.co_freevars)
if func.__closure__ is not None:
print('Their values: ', [x.cell_contents for x in func.__closure__])
else:
print('Their values: []')
reveal(func)Free variables: ('N',)
Their values: [3]
Free variables: ('N',)
Their values: [1]
Free variables: ('N',)
Their values: [2]
Here is a more complicated example to think about:
The following example comes from Python gotchas.
What would you expect from the following code fragment?
def create_multipliers():
multipliers = []
for i in range(5):
def multiplier(x):
return i * x
multipliers.append(multiplier)
return multipliers
for func in create_multipliers():
print(func(2))8
8
8
8
8
What is going on? The inner function multiplier picks up its free variable from the body of the outer function create_multipliers. It picks this up when the function is called, not when it is defined.
This means what matters to the inner function is the most recent value of i at the time it is called.
Here is another example where our reveal shows that the free variables change values if we reassign i.
def create_multipliers():
i = 0
def multiplier(x):
return i * x
reveal(multiplier)
i = 1
reveal(multiplier)
return multiplier
func = create_multipliers()Free variables: ('i',)
Their values: [0]
Free variables: ('i',)
Their values: [1]
A trick to solve this issue would be to store the variable i as a default value for a parameter. Since these values are captured when the function is defined, they will maintain their values:
def create_multipliers():
multipliers = []
for i in range(5):
def multiplier(x, _i=i):
return _i * x
multipliers.append(multiplier)
return multipliers
for func in create_multipliers():
print(func(2))0
2
4
6
8
A nicer solution would be with a class:
class Multiplier:
def __init__(self, n):
self.n = n
def __call__(self, x):
return self.n * x
def create_multipliers():
return [Multiplier(i) for i in range(5)]
for func in create_multipliers():
print(func(2))0
2
4
6
8
Master closures and functions returning functions:
create_calculator(operation) that returns a function performing the specified operation:create_accumulator(initial=0) that returns a function which keeps a running total:memoize function that caches function results:Use the reveal function to understand how closures work in your solutions!
Actually, not quite, since a function can have type arguments and annotations, and these will not be discussed today.↩︎
See this question in the Python FAQ↩︎
The documentation would say we are binding a name to a variable.↩︎
---
title: "Functions Deep Dive"
subtitle: "More than you wanted to know about functions"
author: "Karsten Naert"
date: today
toc: true
execute:
echo: true
output: true
---
# The unpacking operator
You are already familiar with this syntax:
```{python}
my_list = [100, 200]
a, b = my_list
print(a)
print(b)
```
This is a way of _unpacking_ the list into two variables. We could do this, because we know the list has precisely two elements.
Sometimes you don't know how many variables to expect and you just want to grab whatever remains and put it in a list. You can do this with `*name_of_list`. An example:
```{python}
head, *tail = [1, 2, 3, 4]
print(f'{head=}')
print(f'{tail=}')
```
```{python}
*head, tail = [1, 2, 3, 4]
print(f'{head=}')
print(f'{tail=}')
```
```{python}
first, second, *tail = [1, 2, 3, 4]
print(f'{first=}')
print(f'{second=}')
print(f'{tail=}')
```
```{python}
#| error: true
first, second = [1]
print(f'{first=}')
print(f'{second=}')
```
A neat trick is to unpack a list with one element:
```{python}
short_list = [123]
a, = short_list
print(a)
```
Since the star grabs all the rest, it doesn't make sense to do it twice:
```{python}
#| error: true
head, *body, *tail = [1, 2, 3, 4, 5]
```
::: {.callout-note icon=false}
## Exercise
Try to predict what each of these unpacking operations will produce, then test your predictions:
```{python}
#| eval: false
data = [10, 20, 30, 40, 50]
# What do these produce?
a, *b = data
*c, d = data
e, f, *g = data
h, *i, j, k = data
```
Also try unpacking with nested structures:
```{python}
#| eval: false
nested = [[1, 2], [3, 4], [5, 6]]
first, *rest = nested
print(f"First: {first}, Rest: {rest}")
```
:::
More info: read [PEP 3132](https://peps.python.org/pep-3132/)
# Functions from two sides
A function is a piece of code that gets called by another piece of code. That means there are two parties involved:
- the _author_ of the function, the _callee_
- the _user_ of the function, the _caller_
Often you will act as both author and user of a function, but for functions from libraries or builtin functions, these may be different people.
# Calling functions
Calling functions is actually quite a bit simpler than defining them, at least in full generality, so let's start there.
## Positional versus keyword arguments
You already know two ways to pass arguments:
- _positional_, such as `f(1, 2, 3)`. We simply type the variable.
- _by keyword_ , such as `f(a=1, b=2, c=3)`. Here the syntax is `name=variable`.
The one rule here is that keyword arguments must always come after the positional arguments.
We will now see two extra ways to pass arguments: as `args` and `kwargs`.
## args
The `*args` are a way to pass a bunch of arguments directly from a function into a list. Let's say our function looks like this:
```{python}
def f(a, b, c):
return a + b + c
```
And let's say we have a list with precisely three elements
```{python}
the_list = [1, 2, 3]
```
We could call the function for instance like this
```{python}
arg1, arg2, arg3 = the_list
f(arg1, arg2, arg3)
```
But we can also pass the list directly into the function with the unpacking operator:
```{python}
f(*the_list)
```
```{python}
#| error: true
the_list = [1, 2, 3, 4]
f(*the_list)
```
It's perfectly fine to combine this with other arguments, and you can also use it multiple times:
```{python}
the_list = [1, 2]
f(1, *the_list)
```
```{python}
def g(a, b, c, d):
return a + b + c + d
the_list = [1, 10]
g(*the_list, *the_list)
```
A useful way to think about it is that `*` removes the brackets: `g(*[1, 10], *[1, 10])` becomes `g(1, 10, 1, 10)`.
::: {.callout-note icon=false}
## Exercise
Explain the difference in output between these prints:
```{python}
info = ['that', 'all', 'men', 'are', 'created', 'equal']
print(info)
print(*info)
```
:::
## kwargs
For passing multiple keyword arguments at the same time, you can use the syntax `**kwargs` in the same way.
```{python}
def f(a, b, c):
return a + b + c
the_dict = {'a': 1, 'b': 2, 'c': 3}
f(**the_dict)
```
```{python}
the_dict = {'b': 2, 'c': 3}
f(a=1, **the_dict)
```
```{python}
#| error: true
the_dict = {'b': 2, 'c': 3, 'd': 4}
f(a=1, **the_dict)
```
This approach is often used to prepare the arguments in a dict where they can be easily manipulated and then pass them to the function in one go.
```{python}
#| eval: false
import requests
# Build the argument dictionary
kwargs = {
"url": "https://api.github.com/repos/psf/requests",
"headers": {"Accept": "application/vnd.github.v3+json"},
"params": {"per_page": 5}
}
# Call requests.get with the keyword arguments
response = requests.get(**kwargs)
print(response.status_code)
print(response.json())
```
::: {.callout-note icon=false}
## Exercise
Practice using `*args` and `**kwargs` by completing these tasks:
1. Write a function `flexible_sum(*args, **kwargs)` that:
- Sums all positional arguments
- Adds any numeric values from keyword arguments
- Ignores non-numeric keyword arguments
- Returns the total sum
2. Test your function with these calls:
```{python}
#| eval: false
flexible_sum(1, 2, 3) # Should return 6
flexible_sum(1, 2, bonus=5, penalty=2) # Should return 10
flexible_sum(10, name="test", value=3) # Should return 13
```
3. Create a function `debug_call(func, *args, **kwargs)` that:
- Prints the function name and all arguments before calling
- Calls the function with the provided arguments
- Returns the result
Try it with: `debug_call(flexible_sum, 1, 2, bonus=3)`
:::
# Defining functions
Defining functions is actually much harder than using them, because you have to foresee all the ways in which a user will call your function and decide what to do in each case.
The functions structure for how it accepts its arguments (and what type it returns) is often called the _signature_ of the function. By the end of this lecture we will see in full generality how complicated a function signature can get in Python^[Actually, not quite, since a function can have type arguments and annotations, and these will not be discussed today.].
The arguments are often called _parameters_ in the context of defining a function^[See [this question](https://docs.python.org/3/faq/programming.html#what-is-the-difference-between-arguments-and-parameters) in the Python FAQ]. This is because in the function body, they should be considered blanks that are to be filled in by the arguments when the function is called, sometimes combined with default values.
## the vanilla function definition
```{python}
def f(x, y=10, z=20):
print('x =', x, 'y =', y, 'z =', z)
```
This function has three parameters. Each of these can be passed as a position argument and as a keyword argument.
However, two of these parameters have a _default_ value, which implies they can be omitted. Let's look at some examples.
First, we can pass exactly one positional argument. Note that this is ok, because the other parameters have defaults.
```{python}
f(1)
```
Next, we will pass two arguments, both positional. These get grabbed to fill in the parameters x and y, while for the parameter `z` we fall back to its default value:
```{python}
f(1, 2)
```
We can also call this function with purely keyword arguments. In this case the parameters can be provided based on their name:
```{python}
f(y=2, x=1, z=3)
```
We can also mix positional and keyword arguments:
```{python}
f(1, z=3)
```
```{python}
f(1, 2, z=3)
```
It is not allowed to define a parameters twice, typically once with a positional argument and once more with a keyword argument.
```{python}
#| error: true
f(2, x=1)
```
Note that the function caller must always provide the positional arguments first and keyword arguments second, so the following syntax is illegal, regardless of how we define the function:
```{python}
#| error: true
f(x=3, 2)
```
If not all parameters could be found, whether from positional, keyword arguments, or from their default values, we also get an error:
```{python}
#| error: true
f()
```
You can use the following function to investigate the signature of functions. Note that each parameter is indeed `POSITIONAL_OR_KEYWORD`, some with and some without default.
```{python}
import inspect
def reveal_signature(func):
sig = inspect.signature(func)
for name, param in sig.parameters.items():
if param.default is inspect._empty:
print(name, param.kind, '(No default)')
else:
print(name, param.kind, 'default =', param.default)
reveal_signature(f)
```
::: {.callout-note icon=false}
## Exercise
Use the `reveal_signature` function on some functions from third party libraries you know. See if you can find any new sorts of parameters!
:::
::: {.callout-warning icon=true}
The syntax `x=5` means something different from the perspective of the caller versus the callee:
- From the perspective of the caller, it means that you provide a keyword argument named `x` with value 5;
- But from the perspective of the callee, it means that you are providing a default value `5` for this parameter, you are *not* declaring that it must be called as keyword argument.
:::
## variadic positional and keyword-only parameters
You can also define a function that will accept any number of positional arguments by using the unpacking operator. The convention is to use the name `args`.
When the function gets called, args will become a tuple with all the remaining positional arguments that could not be matched.
```{python}
def f(x, *args):
print('x =', x, 'args =', args)
reveal_signature(f)
print()
f(1, 2, 3)
f(1, 2)
f(1)
```
The unpacking operator does not accept arguments passed as keywords, so this would be illegal:
```{python}
#| error: true
f(1, y=2)
```
This also doesn't change the fact that all parameters must eventually receive a value:
```{python}
#| error: true
f()
```
It is possible to combine this with parameters which have a default value, for instance:
```{python}
def f(x, y=10, *args): # <-- weird! don't do this
print('x =', x, 'args =', args, 'y =', y)
reveal_signature(f)
print()
f(1, 2, 3, 4)
f(1, y=5)
f(1)
```
The previous example is legal but a little bit weird. You can provide `y=5` or extra positional arguments `2, 3, 4` but there is no ways to provide both at the same time, because the obvious syntax `f(1, 2, 3, 4, y=10)` actually defines the argument `y` two times, once positional and once with keyword!
What if we did it the other way?
## Positional-only arguments
```{python}
def f(x, *args, y=10):
print('x =', x, 'args =', args, 'y =', y)
reveal_signature(f)
print()
f(1, 2, 3, 4)
f(1, y=5)
f(1)
```
Now something interesting has occurred: no matter how many positional arguments we provide to our function `f`, we will never be able to provide the parameter `y` as a positional argument, because `*args` will eat them all! That makes this a `KEYWORD_ONLY` parameter.
In the previous example our `KEYWORD_ONLY` parameter `y` has a default value, but this is absolutely not required!
```{python}
def f(x, *args, y):
print('x =', x, 'args =', args, 'y =', y)
reveal_signature(f)
print()
f(1, 2, 3, 4, y=10)
f(1, y=5)
f(1, y=10)
```
If there is no default value, and you don't provide one, you get an error of course:
```{python}
#| error: true
f(1, 2, 3)
```
In principle you can give the argument with defaults and without defaults in any order, but it is a little strange to do it, I recommend combining all the parameters with defaults.
```{python}
def f(x, *, a=10, b, c=12, d): # don't do this, prefer: b, d, a=10, c=12
print(f'{x=}, {a=}, {b=}, {c=}, {d=}')
reveal_signature(f)
f(1, a=1, b=2, c=3, d=4)
```
If you want to specify that the parameters are keyword-only but there should not be an unlimited number of positional arguments allowed, you can use this syntax:
```{python}
def f(x, *, z, u=1):
print('x =', x, 'z =', z, 'u =', u)
reveal_signature(f)
f(1, z=1, u=3)
```
The `*` plays the same role as `*args` but it won't eat extra positional arguments.
::: {.callout-tip icon=true}
## Tip
Making all parameters keyword-only is a useful trick! This way you can always change your mind later about the order in which the parameters should appear without breaking existing code.
```{python}
def f(*, x, y):
print('x =', x, 'y =', y)
reveal_signature(f)
print()
f(x=1, y=2)
f(y=2, x=1)
```
```{python}
#| error: true
f(1, 2)
```
:::
## variadic keywords
By analogy with `*args` there also exists `**kwargs` that will eat up all the keyword arguments that could not be matched with any other parameter.
```{python}
def f(y=10, **kwargs):
print('y =', y, 'kwargs =', kwargs)
reveal_signature(f)
print()
f(3, z=10)
f(y=3, z=10)
f(y=3, z=10, u=123, v=1234)
f(z=10, y=3)
f(3)
f(z=10)
```
We can combine this with all the rest:
```{python}
def f(x, u=3, *args, z, y=10, **kwargs):
print('x =', x, 'u=', u, 'args =', args, 'y =', y, 'z =', z, 'kwargs =', kwargs)
reveal_signature(f)
```
## positional-only arguments
Finally, since Python 3.8 we have another syntax that can make arguments positional-only.
```{python}
def f(x, y, /, z):
print('x =', x, 'y =', y, 'z =', z)
reveal_signature(f)
```
```{python}
f(1, 2, 3)
```
```{python}
f(1, 2, z=3)
```
```{python}
#| error: true
f(1, y=2, z=3)
```
This makes this about the most general function signature:
```{python}
def f(a, /, x, u=3, *args, z, y=10, **kwargs):
print('a =', a, 'x =', x, 'u=', u, 'args =', args, 'y =', y, 'z =', z, 'kwargs =', kwargs)
reveal_signature(f)
```
::: {.callout-note icon=false}
## Exercise
Practice with different parameter types by creating these functions and testing them:
1. **Config Function**: Create a function `create_config(name, /, version=1.0, *, debug=False, **options)` that:
- Takes a mandatory positional-only `name` parameter
- Has a `version` parameter with default 1.0
- Has a keyword-only `debug` parameter with default False
- Accepts additional configuration options via `**options`
- Returns a dictionary with all configuration values
2. **Logger Function**: Create `log_message(level, /, message, *details, timestamp=None, **metadata)` that:
- Takes a required positional-only `level` parameter
- Takes a required `message` parameter
- Accepts optional additional details via `*details`
- Has an optional keyword-only `timestamp` parameter
- Accepts metadata via `**metadata`
- Prints all information in a structured format
3. **Test your functions** with various calling patterns:
```{python}
#| eval: false
# Test create_config
config1 = create_config("myapp")
config2 = create_config("myapp", 2.0, debug=True, database="sqlite")
# Test log_message
log_message("INFO", "Application started")
log_message("ERROR", "Connection failed", "timeout after 30s", timestamp="2024-01-01", user="admin")
```
Try invalid calls to understand the error messages!
:::
# Mutable default arguments
The role of the default values of the parameters can sometimes lead to confusion. Consider the following example:
```{python}
def f(x=[]):
x.append(1)
print(x)
f()
f()
```
The first time we call `f`, it starts from and empty list and appends `1`. But the second time, the `1` is already appended and _another_ `1` is appended.
This is because the default value is defined only once, when the function is defined. It is not defined every time the function is called. That means if it gets mutated, all subsequent function calls will see the mutated value. This can be exploited
```{python}
def addition(x, y, _count=[0]): # don't do this
_count[0] += 1
print(f"Addition was called {_count[0]} times.")
return x + y
addition(1, 2)
addition(3, 4)
addition(5, 6)
```
... but there are better ways to achieve the same, with a class or a global variable if you have to.
So what if you actually want an empty list as default argument? Best practice is that you pass `None` and check for it inside the function:
```{python}
def f(x=None):
if x is None:
x = []
...
```
This way you are guaranteed to start from a fresh list every time.
So the lesson to remember is this: default values should not be mutable.
::: {.callout-note icon=false}
## Exercise
Test your understanding of mutable default arguments:
1. **Identify the Problem**: What will this code print? Try to predict before running it:
```{python}
#| eval: false
def append_to_list(item, target=[]):
target.append(item)
return target
print("First call:", append_to_list("a"))
print("Second call:", append_to_list("b"))
print("Third call:", append_to_list("c"))
```
2. **Fix the Function**: Rewrite the function above to avoid the mutable default argument problem.
3. **Advanced Challenge**: Create a function `create_counter(start=0)` that returns a function which increments and returns a counter. Each call to `create_counter()` should create an independent counter:
```{python}
#| eval: false
counter1 = create_counter()
counter2 = create_counter(10)
print(counter1()) # Should print 1
print(counter1()) # Should print 2
print(counter2()) # Should print 11
print(counter1()) # Should print 3
```
4. **Debugging Exercise**: Explain why this "clever" attempt to solve the mutable default problem doesn't work:
```{python}
#| eval: false
def broken_fix(item, target=[].copy()):
target.append(item)
return target
```
:::
# Scope and oddities
## variable scopes
If you read a bit in the official Python documentation about the [execution model](https://docs.python.org/3/reference/executionmodel.html) you will learn that Python code is built out of _blocks_. There are many types of blocks but the most important ones are:
- a module (a Python file, for our purposes)
- a class definition
- a function definition
If you define a variable, it comes with a _scope_ attached, this is the piece of code where the variable is visible from. The scope is usually just the same block, but for function blocks it is also the blocks contained in it.
To _look up_ a variable, Python simply looks at the surrounding scope. In the next example we look up a variable `y`. The function `f` looks it up from it surrounding block:
```{python}
y = 10
def f():
return y
f(), y
```
When **defining** the function `f`, it registers that it needs to return the global variable `y`.
If we change the value of the variable after the function is defined, we will receive the adapted value. The function `f()` looks this value up in its higher scope.
```{python}
y = 10
def f():
return y
y = 20
f(), y
```
::: {.callout-warning icon=true}
We do _not_ capture the _value_ of `y` when the function is defined. That means if `y` changes value before the function is called, `f` will still look up its latest value.
:::
The situation is very different when we _assign_ a value to `y` inside the function^[The documentation would say we are binding a name to a variable.]. In the next example another, second variable `y` is created, which only exists inside the inner scope. So the example has two variable, with the same name but a different scope:
```{python}
y = 10
def f():
y = 1
return y
y = 20
f(), y
```
The command `return y` gives us the value of the local variable `y` and ignores the global `y`.
So how does Python decide whether to return the local or global variable? The rule is: if the variable is assigned, or technically, if the name is bound, **anywhere** in a block, that variable is considered local.
Consider this example:
```{python}
#| error: true
y = 10
def f():
if False:
y = 1
return y
y = 20
f(), y
```
Because `y` is defined inside the block `f`, the variable is considered local. It does not matter that we hit this line of code or not! So when we return `y`, Python tries to return a local variable that has not received a value yet. Technically, the name of the local variable has not been bound to a value. The result is an ugly `UnboundLocalError`.
## local and global
If we don't want Python to consider a variable as local, the command `global y` will change this. Here, the same thing is true: if Python finds this **anywhere** in the function, it will treat all occurrences of `y` as global. This can overrule the earlier rules, which allows you to change a global variable from within a function after all.
```{python}
y = 10
def f():
global y
y = 15
return y
y = 20
f(), y
```
What matters is that you declare `y` to be global. It doesn't matter that you hit this line or code or not.
```{python}
y = 10
def f():
if False:
global y
y = 15 # this would make y a local variable
# but it is overruled by the global y
return y
f(), y
```
::: {.callout-note icon=false}
## Exercise
Explain the outcome of the following code:
```{python}
#| eval: false
y = 10
def f():
y += 1
return y
f()
```
:::
## Functions that return functions
A function can return another function:
```{python}
def add_N(N):
def add(x):
return x + N
return add
func = add_N(3)
func(4), func(5)
```
The variable `N` is a free variable inside the function `add`. The function can then look up its value.
Here is some helper code to understand how this works:
```{python}
def reveal(func):
print('Free variables: ', func.__code__.co_freevars)
if func.__closure__ is not None:
print('Their values: ', [x.cell_contents for x in func.__closure__])
else:
print('Their values: []')
reveal(func)
```
```{python}
func = add_N(1)
func(4), func(5)
```
```{python}
reveal(func)
```
```{python}
func1 = add_N(1)
func2 = add_N(2)
reveal(func1)
reveal(func2)
```
Here is a more complicated example to think about:
```{python}
def plus_N_times_M(M):
def add_N(N):
def add(x):
return M * (x + N)
return add
return add_N
plus_N_times_M(100)(3)(1)
```
```{python}
reveal(plus_N_times_M(100)(3))
```
```{python}
reveal(plus_N_times_M(100))
```
## A more elaborate example
The following example comes from [Python gotchas](https://docs.python-guide.org/writing/gotchas/#late-binding-closures).
What would you expect from the following code fragment?
```{python}
def create_multipliers():
multipliers = []
for i in range(5):
def multiplier(x):
return i * x
multipliers.append(multiplier)
return multipliers
for func in create_multipliers():
print(func(2))
```
What is going on? The inner function `multiplier` picks up its free variable from the body of the outer function `create_multipliers`. It picks this up when the function is called, not when it is defined.
This means what matters to the inner function is the most recent value of `i` at the time it is called.
Here is another example where our reveal shows that the free variables change values if we reassign `i`.
```{python}
def create_multipliers():
i = 0
def multiplier(x):
return i * x
reveal(multiplier)
i = 1
reveal(multiplier)
return multiplier
func = create_multipliers()
```
A trick to solve this issue would be to store the variable `i` as a default value for a parameter. Since these values are captured when the function is defined, they will maintain their values:
```{python}
def create_multipliers():
multipliers = []
for i in range(5):
def multiplier(x, _i=i):
return _i * x
multipliers.append(multiplier)
return multipliers
for func in create_multipliers():
print(func(2))
```
A nicer solution would be with a class:
```{python}
class Multiplier:
def __init__(self, n):
self.n = n
def __call__(self, x):
return self.n * x
def create_multipliers():
return [Multiplier(i) for i in range(5)]
for func in create_multipliers():
print(func(2))
```
::: {.callout-note icon=false}
## Exercise
Master closures and functions returning functions:
1. **Calculator Factory**: Create a function `create_calculator(operation)` that returns a function performing the specified operation:
```{python}
#| eval: false
add = create_calculator("add")
multiply = create_calculator("multiply")
print(add(5, 3)) # Should print 8
print(multiply(4, 6)) # Should print 24
```
2. **Accumulator Pattern**: Create a function `create_accumulator(initial=0)` that returns a function which keeps a running total:
```{python}
#| eval: false
acc1 = create_accumulator()
acc2 = create_accumulator(100)
print(acc1(5)) # Should print 5
print(acc1(10)) # Should print 15
print(acc2(5)) # Should print 105
print(acc1(2)) # Should print 17
```
3. **Fix the Late Binding Bug**: Explain why this code doesn't work as expected, then fix it:
```{python}
#| eval: false
def create_functions():
functions = []
for i in [1, 2, 3]:
def func():
return i * 10
functions.append(func)
return functions
# This prints the wrong values - why?
for f in create_functions():
print(f())
```
4. **Advanced Challenge**: Create a `memoize` function that caches function results:
```{python}
#| eval: false
def memoize(func):
# Your implementation here
pass
# Should cache results of expensive calculations
@memoize
def fibonacci(n):
if n < 2:
return n
return fibonacci(n-1) + fibonacci(n-2)
print(fibonacci(10)) # Should be fast on subsequent calls
```
Use the `reveal` function to understand how closures work in your solutions!
:::
# Related topics for another day
- lambdas
- exec and eval
- the functools library
- type hints
- decorators
# Additional resources
- [Python Official Documentation - Functions](https://docs.python.org/3/reference/compound_stmts.html#function-definitions)
- [Python Glossary - Definition of Parameter](https://docs.python.org/3/glossary.html#term-parameter)
- [Mutable default arguments](https://docs.python-guide.org/writing/gotchas/#mutable-default-arguments)