The Lean Python Blog - Python

Assert Assertions

2024-05-11T13:00:00+12:00

What has your experience of assert been? Do you agree with this post? Do you disagree? Please comment below.

Introduction

The assert statement enables us to state something which we require to be true. We can also include a comment to display should the assertion fail. For example:

def get_square(num: float) -> float:
    assert num > 0, f"num should be greater than 0 but was {num:,}"
    square = num ** 2
    print(f"The square of {num} is {square:,}")
    return square

get_square(12)
get_square(-3)
get_square(666)

In this case, as long as asserts haven't been deactivated (through the use of the -O optimise option), the program will halt when we pass -3 to the get_square function.

The square of 12 is 144
...
AssertionError: num should be greater than 0 but was -3

If we run our script with the -O option then the program will accept the -3 value and output the following:

The square of 12 is 144
The square of -3 is 9
The square of 666 is 443,556

(Simplified) Guidance

1) With rare exceptions we should only use asserts in unit tests e.g. when using pytest

2) We shouldn't use asserts as comments or documentation - we should write actual comments in readable English

3) If we want to program defensively then we should use exceptions rather than asserts (which can be just switched off, even if this is rare in practice)

4) We shouldn't use asserts to check for (currently) impossible values. So instead of:

if value == MyEnum.SPAM:
   ...
elif value == MyEnum.EGG:
   ...
else:
    assert False, f"Unexpected {value=}" # This can never happen if we've checked all the cases in the Enum

we should write:

if value == MyEnum.SPAM:
   ...
elif value == MyEnum.EGG:
   ...
else:
    raise ValueError(f"Unexpected {value=}") # This can never happen if we've checked all the cases in the Enum

Why? Because it allows us to simplify the basic guidance on using assert - all things being equal we should reduce the complexity of our code and the things we have to learn or teach. Furthermore, there is no point worrying about the performance cost of an exception vs an assert (assuming it is deactivated using -O flag) in this case - the exception will presumably only ever be raised once before the bug is found and fixed.

5) There are special cases where we might want to use assertions outside of unit tests. Simple guidance doesn't have to cover these. Examples of special cases that can be ignored in simple / simplified guidance:

While teaching how to program so students can self-test their code with asserts you provide
To assist type checkers handle unreachable code. Note - we could just use an exception there as well.
Using something like assert my_value is not None etc to keep mypy happy
Avoiding the overhead of exception raising in extremely performance-sensitive code (although why are we using Python in that case?)
When debugging code

Warning

Don't wrap assert in parentheses if you have a comment. Surprisingly, they cease to operate! For example:

assert 1 == 0  ## will fail
assert (1 == 0)  ## will fail
assert 1 == 0, "1 doesn't equal 0"  ## will fail
## but
assert(1 == 0, "1 doesn't equal 0")  ## will pass

Conclusion

There will be plenty of experienced developers who use assert to improve their Python code. And there is nothing wrong with that. But the best (simple) advice from a Lean Python point of view is to restrict the use of assert to testing and use exceptions and comments instead.

String Theory

2024-04-21T07:00:00+12:00

What has your experience of working with strings, especially f-strings, been? Do you agree with this post? Do you disagree? Please comment below.

Concatenation, Gluing, and Interpolation

Concatenation is one of the first ways we learn of joining strings together in Python, and it remains useful for simple cases, even when we learn about more sophisticated alternatives:

fname = 'Jo'
print('Hello ' + fname)

'Hello Jo'

Glueing, or joining strings together with a glue string, is especially useful when working with a sequence of items we wish to present together:

print(' \N{WHITE HEART} '.join(['Romeo', 'Juliet']))

'Romeo 🤍 Juliet'

In version 3.6, Python introduced formatted string literals, better known as f-strings. These make it possible to inject, or interpolate, other strings inside a larger string(¹). f-strings are more readable than the alternative approaches Python had used earlier - namely sprint and the format method.

## sprint
fname = 'Elon'
print("Hello %s" % fname)

'Hello Elon'

lname = 'Musk'
print("Hello %s %s" % (fname, lname))

'Hello Elon Musk'

## format
print("Hello {fname} {lname}".format(fname=fname, lname=lname))
print("{unit_id:0>10}".format(unit_id="ABC1234"))  ## just showing how powerful the format method can become

'Hello Elon Musk'

'000ABC1234'

## f-string
fname = 'Elon'
lname = 'Musk'
print(f"{fname} {lname}")

'Elon Musk'

When Not to Use f-string Interpolation

The first question is whether interpolation should be used at all. Sometimes there is a more elegant solution using string methods.

word = 'crocodile'

## f-string overkill
print(f"{word:*^30}")

## semantic string method
print(word.center(30, '*'))

'**********crocodile***********'

In other cases interpolation of any sort is not a good idea from a security point of view. If strings are being built from variables which in any way come from user input they open up vulnerabilities like SQL injection attacks. Here is the warning from the older psycopg docs: “Never, never, NEVER use Python string concatenation (+) or string parameters interpolation (%) to pass variables to a SQL query string. Not even at gunpoint.” And in newer documentation it says:

Don’t manually merge values to a query: hackers from a foreign country will break into your computer and steal not only your disks, but also your cds, leaving you only with the three most embarrassing records you ever bought. On cassette tapes. If you use the % operator to merge values to a query, con artists will seduce your cat, who will run away taking your credit card and your sunglasses with them. If you use + to merge a textual value to a string, bad guys in balaclava will find their way to your fridge, drink all your beer, and leave your toilet seat up and your toilet paper in the wrong orientation. You don’t want to manually merge values to a query: use the provided methods instead. Passing parameters to SQL queries

Even if interpolation is appropriate, it will sometimes be best to use an alternative to f-strings. The main situation in which it makes sense to use sprint or the format method instead is when the main string contains lots of curly braces and it is inefficient to escape all of them. And template strings might be best for untrusted user input Python String Formatting Best Practices

f-string Tips (Format Strings)

f-strings have a mini formatting language. Here are four useful tricks (for more, see Python F-String Codes I Use Every Day ):

Zero-Padding

a = 12345
print(f"{a:0>10}")

'0000012345'

Commas in Numbers

print(f"{a:,}")

'12,345'

Percentages

num = 0.5
print(f"{num:.0%}")

'50%'

Dates

from datetime import datetime
print(f"{datetime.today():%Y-%m-%d}")

'2024-04-21'

or expressions resulting in strings, or anything which has a __str__ method ↩

Dataclasses Considered Sweet

2024-04-08T12:50:00+12:00

Dataclasses Considered Sweet

Overview

Dataclasses are versatile and they're a crucial part of Python.

Firstly, dataclasses are a great data structure. They enable us to store multiple values in a customised and clearly defined structure. Alongside lists and dictionaries, dataclasses are a valuable fundamental building block in our Python programs.

Secondly, dataclasses are a super-convenient way of defining (most) classes, not just data classes. In addition to removing a whole lot of boilerplate every time, they provide a very readable definition of the (instance) attributes in a class.

These benefits do come with a cost though, and these costs will need to be mitigated.

There will also need to be guidance on when to use a dataclass and when to use a standard class.

Dataclasses as Foundational Data Structures

The more complex a program, the more we benefit from making the meaning of the different parts of the program explicit. We do this by giving things names. So we often shift from data structures that are positional only to those that rely on keywords. We might shift, for example, from tuples to named tuples, or from lists to dictionaries. This is where dataclasses really shine. Even though dataclasses can do much, much more, in their simplest version they provide a very concise, readable syntax for defining a data structure that names its internal parts and, at a basic level, documents them.

For example:

@dataclass
class Coord:
    lbl: str
    lat: float
    lon: float
    srid: int = 4326

There is very little boilerplate and it is hard to think of a more concise way of defining attributes. If we get an instance of a Coord class in our program we know exactly what we're getting and we can refer to parts of it by name. We can also look up the definition of the dataclass to see if there is extra documentation. I recently saw a student using a list as a data structure for configuration and his code became increasingly confusing as he changed the number, type, and order of the items. Items were read from the list by position and it was therefore difficult to read the code for correctness.

As mentioned in the Overview, dataclasses can be used as a convenient way of making classes in general, not just data-centric classes. But even when we are using our dataclass exclusively as a data structure, there are lots of general class features we can add that make the data structure safer, and more convenient to use. Note - dataclasses are worth using as data structures even when kept very simple - don't be deterred by all the extra things you need to understand to strengthen your dataclass.

We can add derived values as properties - for example:

@dataclass
class Area:
    h_metres: float
    w_metres: float
    lbl: str | None = None

    @property
    def h_cm(self):
        return self.h_metres * 100

    @property
    def area_sq_metres(self):
        return self.h_metres * self.w_metres

We can add validation using the dataclass-supported post_init method:

    def __post_init__(self):
        if not 1 <= self.h_metres <= 100:
            raise ValueError(f"h_metres value ({self.h_metres}) not in expected range")

We can make a nice representation of dataclass objects when printed e.g. in log files or error messages:

    def __str__(self):
      return f"Area {self.h_metres}m x {self.w_metres}m ({self.area_sq_metres:,} square metres)"

We can potentially handle default values. Depending on what we are trying to do there are different strategies. In ascending order of trickiness we have:

1) equals e.g.

@dataclass
class Business:
    ...
    country: str = 'NZ'

2) using post_init, where we have access to all the values in the dataclass (including any defined by property) e.g.

    def __post_init__(self):
        if self.lbl is None:
            self.lbl = f"{self.name} (auto-generated)"

This seems to be the best (only?) way of working with other instance attribute values. See https://stackoverflow.com/questions/66437553/reference-a-variable-of-a-dataclass-in-a-field-with-a-default-factory

3) default_factory for a fresh mutable (one that isn't shared by all instances of the class)

from dataclasses import dataclass, field

class Business:
    ...
    branches: list = field(default_factory=list)

4) default_factory for a function

from uuid import uuid4

class Business:
    ...
    user_id: str = field(default_factory = lambda : str(uuid4()))

Finally, we can add convenience functions for ingesting or exporting data - for example, in the case of a Coord object we might make it easy to write something like:

new_coord = coord.with_new_srid(srid=2193)

Whether we make a simple dataclass, or a more sophisticated one, having a well-named data structure makes it much easier to define interface expectations in the program. Basically, what does a function require as an argument and what will it return as a result. Dataclasses make it easy to just use a name in type hinting that is self-explanatory e.g. without even having to look at the code we might reasonably infer that an instance of an OutputSpec dataclass defines the configuration for our output. And if we want to know what aspects of the output are configured we can just look at the dataclass definition for OutputSpec.

Dataclasses as Syntactic Sugar

Dataclasses provide some syntactic sugar by replacing:

class Coord:

    def __init__(self, lbl: str, lat: float, lon: float, srid: int = 4326):
        self.lbl = lbl
        self.lat = lat
        self.lon = lon
        self.srid = srid

with:

@dataclass
class Coord:
    lbl: str
    lat: float
    lon: float
    srid: int = 4326

The best bit is the removal of all the repetitive boilerplate self.x = x etc. And the code is more declarative - in most cases it is very easy to see what attributes will be in the class.

Sweet Enough?

There is a dark side to dataclasses though - the dataclass decorator makes a series of "magic" changes to your standard class - not least of all the behind-the-scenes transformation of class attributes into instance attributes. The main downside is that people might become confused about the syntax for defining class versus instance attributes and accidentally add class attributes to standard classes. This is already happening in the wild. And Python generally favours explicit over implicit as per The Zen of Python.

The question is - do the benefits of dataclasses outweigh the costs? In the case of data-centric classes, where classes are used as data structures alongside dictionaries and lists and so on, the answer is Yes. In practice, the declarative nature of the dataclass, the ability to flexibly handle default values using fields etc, is so valuable it is worth potential confusion about ways of declaring class versus instance attributes. But what about other uses of dataclasses? Is it worth the rather modest convenience of reducing boilerplate?

To be honest, I'm not sure. My suspicion is that the syntactic sugar of the dataclass is sufficiently sweet that we'll see lots of uses in the wild beyond usage as a data structure. I have already encountered one dev team that mainly uses dataclasses for all class use. In short, the community will decide. But a practical rule-of-thumb might be - use a dataclass unless you find yourself overriding lots of the magic, in which case, a standard Python class will be more explicit. Hacking something that is already magic sounds like a recipe for mysterious behaviours, increased mental load, and subtle bugs.

On a final note, if people are commonly going to use dataclasses as generic classes, it is especially important that Python teachers and resources clarify the difference between a class attribute and and instance attribute. Different languages handle this in different ways which only adds to the potential for confusion.

Normal class:

class Business:
    counter = 0  ## <======= class attribute shared by all instances

    def __init__(self, name: str):
        self.name = name  ## <======= instance attribute unique to an instance

@dataclass
class Business:
    name: str  ## <======= instance attribute (once decorator does its magic)

The Power of Sane Defaults and Python Packaging

2024-03-07T07:00:00+13:00

What has your experience of Python packaging been? Do you agree with this post? Do you disagree? Please comment below.

The When of Python and Sane Defaults

Programming requires choice after choice after choice. It can be draining. Anything simplifying the choices we have to make is, all things being equal, a Good Thing. We can then focus on expressing our ideas in code and solving the problems we're there to solve.

The When of Python is about reducing the mental load of choice when it comes to Python language features. Should we use match case or if else? Should we use a for loop or a comprehension? Are there some features we can ignore or almost always ignore? What is a sane default for most cases?

Annointing a Sane Default for Python Packaging

Which brings us to Python packaging. The community needs to get behind a single sane default that can serve beginners as well as advanced users (Python Dependencies Are Fixable). Perhaps Rye / UV will provide that sane default for packaging. It seems to have built successfully on the hard work and the thinking that has gone into earlier packaging solutions. Although the present state of Python packaging is a mess¹ (try teaching Python packaging to a Python beginner if you don't believe me) it is not because people in the Python community have been making terrible decisions. It has taken a while to get to the point where better things are possible. There is good reason to think Rye / UV will be that better thing.

The time for waiting patiently for one packaging tool to gradually and definitively rise above the others is over. The community, which is expanding rapidly as new people learn Python to start using the data analytics and AI tools, needs a sane default now (see Andrew Ng calls out Python package management as a bottleneck to AI development). This is not to say that today's sane default will last forever, but that a tool like Rye / UV is arguably good enough right now for both beginners and those with advanced / specialist needs (see Is UV the FUTURE of Python PACKAGING? Hynek Schlawack and Python Packaging Features Venn Diagram). We should just annoint it as the Sane Default and start focusing on it when teaching beginners or starting new projects. Even if there might be legitimate reasons to use other packaging approaches in specific cases that doesn't undermine the importance of there being a Sane Default. It would be great to make this a solved problem for what is arguably the world's most important programming language.

Obligatory XKCD reference Python Environment ↩

Simple Async

2023-09-24T14:00:00+13:00

Asynchronous Programming

Being able to achieve tasks at the same time (asynchronously) instead of one-after-the-other (synchronously) can significantly speed up computer processes. The total time for a set of tasks can be the duration of the slowest task (plus a little overhead) rather than the sum of all the tasks combined. Some languages are built for asynchronous programming from the ground up but even though that is not the case with Python it can be surprisingly easy to run concurrent tasks.

Alternatives

Different Python libraries for providing asynchronous programming often have radically different approaches. So while there may not be One Obvious Way for asynchronous programming in general there might be One Obvious Way for each approach. Each has its own strengths and weaknesses and in many cases it is a matter of personal taste¹ as to which one we use. There are numerous articles on the topic but two of the best are:

Reach for Pool Executors First

As recommended in "The easy way to concurrency and parallelism with Python stdlib" (above), we should reach for the pool executors first and only use more complicated approaches if we have special requirements.

"You can distribute work to a bunch of process workers or thread workers with a few lines of code:

from concurrent.futures import ThreadPoolExecutor, as_completed

with ThreadPoolExecutor(max_workers=5) as executor:
    executor.submit(do_something_blockint)

This simple approach only covers a limited use cases, but for those, it works surprisingly well. What's more, those use cases are the ones you are most like to have to solve by yourselves, as the other ones often have infra or libs solutions already."

Simple Pool Executor Recipes

The following talk was delivered several years ago but remains up-to-date - check it out - it contains multiple simple recipes using the pool executors supplied by concurrent futures.

Practical Python Async for Dummies

People sometimes express strong preferences - in one comment, a programmer memorably described asyncio as a "complex inefficient single-core evil twin of a language" Comment on Asyncio article ↩

The Challenge of Simplicity

2023-09-22T08:50:00+12:00

The Ongoing Spread of Python

Python continues to grow in popularity and it is quite possible that Python will eventually be programming. In my Kiwi PyCon 2023 lightning talk I predicted that this will be the case by 2045 i.e. in about two decades. Not in the sense that other languages won't exist, or that other languages won't be important - but in the sense that nearly all programmers will know Python in addition to any other languages. The assumption behind this prediction is that Python will be aggressively refined and improved as it becomes more important and that these refinements will further increase adoption.

Should this prediction be correct, Python, including Python derivatives such as Mojo and PyPy (and new languages based on Python which don't even exist yet), will be used for even more situations. And these situations will be complex in new ways. As Christopher Neugebauer pointed out in his Kiwi Pycon talk "The Complexity of Simplicity", we can take complexity as given so the only question is where it gets handled. Either Python requires users to build complex tools out of simple components, or it provides simple constructs that fit specific situations e.g. matrix multiplication, or asynchronous processing.

Everything Needs a Simple Interface

The question is whether the new interfaces require any changes to the core language or not. An example of a change to the core language is the addition in 2015 of the "@" operator. The operator was added in Python 3.5 to enable matrix multiplication PEP 465 - A dedicated infix operator for matrix multiplication. But these cases are rare and the situations for which they are appropriate, and inappropriate, should be clearly communicated when they are introduced. Much more common is the creation of specialist libraries with their own syntax e.g. pandas, polars etc. These can come and go while the core language remains unaffected. So there is no reason to expect that Common Python will have to expand significantly to cope with increasing situational demands.

There Will be More Python

The end result will possibly be a core, Common Python very similar to what we have now; an ecosystem of Python-like languages (some optimised for speed); and a large array of specialist libraries covering almost every imaginable programming task. No individual will be able to understand all of it. But we should all be able to understand Common Python and the Situational Python we regularly use.

It doesn't seem out-of-order to suggest that we slim down and refine Common Python just because there will be an ever-expanding Situational Python. Admittedly, maintaining simplicity at the same time as adding new complexity will require self-discipline but Python has a good track record in this regard. While staying simple isn't always simple, it is a realistic goal.

Deprecation Appreciation

2023-09-02T19:00:00+12:00

Python as a Designed Language

One of the things I liked when switching from PHP to Python was the way a little learning went further. Once I understood how to perform certain operations with strings e.g. slicing, I also automatically knew how to do the same thing with other sequences. Python basically felt more elegant and designed. Less was more. And it didn't feel like Python had just grown and grown - it felt like it had been designed.

The "designed" feeling is not just about what has been included but also about what is left out. Sometimes different ways of doing the same thing are noise making it harder to learn any of the alternatives properly. As it says in the Zen of Python, "There should be one-- and preferably only one --obvious way to do it."

Reality

In reality, Python has not been able to retain all the elegance of its earliest, smaller and less capable versions. Some good features have been added without others being deprecated, or, at least, explicitly reserved for more specialised use cases - see Classy Data with Dataclasses. Over time, this situation only gets worse unless there is some process of deprecation.

Deprecation

Fortunately, the deprecation process seems to be ramping up, at least for modules. See PEP 594 – Removing dead batteries from the standard library for a discussion of hard deprecations and the planned timetable.¹ There is also a soft deprecation process Formalize the concept of “soft deprecation” (don’t schedule removal) in PEP 387 “Backwards Compatibility Policy”. Soft deprecation is good because it can be exercised more quickly and widely than hard deprecation - there won't be as many unintended consequences or side effects. In general, Python is adopting a much more considered approach to deprecation than in the past - see Python Incompatible Changes - Victor Stinner

Deprecated Python

The When Of Python splits Python features into Common Python, Situational Python, and Deprecated Python. Deprecated Python is an especially useful concept because it allows us to shrink Python and better meet the goal of having one obvious way of doing things. Ideally we will deprecate more features in future releases of Python.

Sometimes there can be a stay of execution. For example, David Beazley argued that the wave module is "easy to teach to kids and can make crazy sounds". According to the PEP, therefore, "it's a fun battery to keep." ↩

Restrict Struct?

2023-07-22T10:00:00+12:00

What has your experience of structs in other languages been? Do you agree with this post? Do you disagree? Please comment below.

Introduction

Brett Cannon recently proposed adding a struct syntax for Python Proposing a struct syntax for Python.

struct Point(x: int, y: int)

Like all his articles it was a fascinating read. And you have to respect someone who talks about Python with friends while doing a vacation road trip :-). But we should strongly resist adding yet more increasingly specialised features to the core language. So let's briefly look at some of the issues raised by the article (directly or indirectly) and possible responses from a When-Of-Python point of view.

Some Arguments

We need an alternative to namedtuple and dataclasses

There probably are use cases where struct is better than dataclass but they aren't important enough to add yet another language feature to the core language.

In practice you'd end up needing to know dataclasses anyway because sometimes you need mutability. So now you have to teach both rather than teaching just the one and teaching it well.

But we're agreed that it would be a good idea to remove collections.namedtuple.

We don't need methods on our data structures

A lot of the time we don't need methods in our data structures. And having methods does add an overhead. But methods are often very useful, especially when wanting to pre-calculate derived values or to validate input (in dataclasses most commonly using the post_init method).

And because we will sometimes need dataclasses as well as something like a struct, we should probably just use dataclasses in both cases (with rare exceptions). The same principle applies in cooking. Do you really want to clutter up your kitchen with unitasker gadgets like meat claws and egg cubers. The 'Unitasker' Kitchen Gadgets Alton Brown Loves To Loathe

Struct has an elegant syntax

Agreed - when there are only a few attributes:

struct Point(x: int, y: int)

But I often find my dataclasses gain additional useful attributes over time and the vertical layout of dataclasses proves really readable - especially when we want type hints, default arguments, and comments:

@dataclass
class Point:
    x: int
    y: int

@dataclass
class Point:
    x: int
    y: int
    srid: int = 4326  # WGS84

I also like how you can incrementally boost the robustness as you feel the need e.g.

@dataclass
class Point:
    x: int
    y: int
    srid: int = 4326  # WGS84

    def __post_init__(self):
        if not 173 <= self.x <= 174:
            raise ValueError(f"Out of range x value {self.x}")

and add new attributes in a logical order without having to thinking about the order of defaults:

@dataclass(kw_args=True)
class Point:
    lbl: str = 'Unknown'
    x: int
    y: int
    srid: int = 4326  # WGS84

    def __post_init__(self):
        if not 173 <= self.x <= 174:
            raise ValueError(f"Out of range x value {self.x}")

Brett acknowledges the problem with adding documentation to structs:

I wish there was a way to do native docstring support while keeping this a single line, but e.g., struct Point(x: int, y: int), "a 2D point." just doesn't look right to me. Since it is a new keyword it might be possible to make a : optional and only usable to add a docstring, but that might be a little too weird when the : doesn't alow for other statements afterwards. Otherwise doing a bare string immediately after the definition could inherently be picked up as a docstring just like what PEP 257 calls an "attribute docstring".

Documentation is another thing dataclasses make easy and documentation really matters:

@dataclass
class Point:
    """
    Point for inclusion in standard plots.
    Has validation for country.
    """
    x: int
    y: int

Optional typing

To be very specific as to why I think this could be better than dataclasses: Typing is optional (there are still folks who don't want to lean into that and it simply isn't always necessary)

In practice I usually have very primitive data types in my dataclasses like str or int (or other dataclasses or enum.StrEnums) but even if you don't want to think about typing, dataclasses accept ... and typing.Any e.g.

@dataclass
class Point:
    x: ...
    y: ...

More details and options can be found at death and gravity: Dataclasses without type annotations

Performance

Class construction should be faster (which intuitively you would think shouldn't matter, but I have talked to folks where this is an actual concern, especially when startup time is critical)

In cases where raw performance really matters we should consider another solution rather than adding something to the core Python language which we'll then have to teach as well as all the existing content.

Misc

Syntax typically leads to better tooling support since there's no ambiguity

True - especially with tools like mypy - but those tools will be forced to handle dataclasses well at some point so ...

Easier to teach than (data)classes, so can act as a stepping stone towards classes

But you're going to have to teach dataclasses anyway so why add teaching structs to that as well.

Better semantics than dataclasses have by default (at least in my opinion 😁)

Maybe, maybe not - but people will have to learn both so better to become confident with one - and then really learn it well.

Conclusion

Adding struct to Python is tempting because structs would be better than dataclasses in particular ways. But they would be Yet Another Thing To Learn and Teach. And they aren't worth it in the core language. It is like cluttering up your kitchen with unitasker gadgets like meat claws and egg cubers.

Match Case - A Simple Switch Statement for Python?

2022-10-05T07:00:00+13:00

What has your experience of match case (Structural Pattern Matching) been? Do you agree with this post? Do you disagree? Please comment below.

Simple Switch Statement at Last?

Python has always lacked a switch statement to perform conditional logic e.g.

## Note - not actual Python
case foo:
    when eggs:
        return 'spam'
    when knights:
        return 'Ni!'
    ...

The switch approach can be found in languages like PHP and also in SQL (CASE WHEN THEN END etc).

Instead, Python has relied largely on if-elif-else constructs and, to a lesser extent, dictionaries - for example:

mapper = {'eggs': 'spam', 'knights': 'Ni!', ...}
result = mapper['eggs']

In Python 3.10, Structural Pattern Matching (match case) was added which raised some questions. Was this basically a switch statement on steroids? Should we all start using this for our conditional logic? And could it be time to largely retire if-elif-else constructs? Initial impressions were positive. Certainly the higher level syntax for Structural Pattern Matching looks clean and elegant:

match food:
    case 'eggs':
        return 'spam'
    case ...

Unfortunately major concerns arise when we take a closer look.

Python Pattern Matching Problems

New Mini Language

Structural Pattern Matching is a new mini language in Python - a powerful language that looks like normal Python but isn't. And that is where the main gotchas come from.

Similarity to Object Instantiation Misleading

Imagine we have a Point class:

class Point:
    def __init__(self, x, y):
        pass

case Point(x, y): seems to me to be an obvious way of looking for such a Point object and unpacking its values into x and y but it isn't allowed. It is a perfectly valid syntax for instantiating a Point object but we are not instantiating an object and supplying the object the case condition – instead we are supplying a pattern to be matched and unpacked. We have to have a firm grasp on the notion that Python patterns are not objects. If we forget we get a TypeError:

case Point(0, 999):

## TypeError: Point() accepts 0 positional sub-patterns (2 given)

Note, we must match the parameter names (the left side) but can unpack to any variable names we like (the right side). For example, all of the following will work:

case Point(x=x, y=y):
case Point(x=lon, y=lat):
case Point(x=apple, y=banana):

but

case Point(a=x, b=y):

will not.

It is a bit disconcerting being forced to use what feel like keyword arguments in our patterns when the original class definition is optionally positional. We should expect lots of mistakes here and it will require concentration to pick them up in code review.

Similarity to `isinstance` Misleading

case Point:, case int:, case str:, case float: don’t work as you might expect. The proper approach is to supply parentheses: using the example of integer patterns, we need case int():, or, if we want to "capture" the value into, say, x, case int(x):. But if we don't know about the need for parentheses, or we slip up (easy to do) these inadvertant patterns will match anything and assign it to the name Point or int or str etc. Definitely NOT what we want.

The only protection against making this mistake is when you accidentally do this before other case conditions – e.g.

case int:
^
SyntaxError: name capture 'int' makes remaining patterns unreachable

Otherwise you are on your own and completely free to make broken code. This will probably be a common error because of our experience with isinstance where we supply the type e.g. isinstance(x, int). Which reminds me of a passage in Through the Looking-Glass, and What Alice Found There by Lewis Carroll.

‘Crawling at your feet,’ said the Gnat (Alice drew her feet back in some alarm), ‘you may observe a Bread-and-Butterfly. Its wings are thin slices of Bread-and-butter, its body is a crust, and its head is a lump of sugar.’

‘And what does it live on?’

‘Weak tea with cream in it.’

A new difficulty came into Alice’s head. ‘Supposing it couldn’t find any?’ she suggested.

‘Then it would die, of course.’

‘But that must happen very often,’ Alice remarked thoughtfully.

‘It always happens,’ said the Gnat. After this, Alice was silent for a minute or two, pondering.

Use Cases (in Python)

OK - so we should probably avoid case match in any situation where it is simpler and safer to use if-elif-else or another alternative - which will almost be all cases. Although Pattern Matching might be a great feature in Scala, it doesn't appear likely to be all that useful in a language like Python.

Having said that, match case is not completely without value for Python. Structural Pattern Matching does appear to be useful for parsing complex nested data such as ASTs or deeply nested JSON. ¹

I do not recommend embracing match-case EXCEPT in the unusual cases where it's VERY handy.

The use case I had happened to be one of those weird cases (recursively matching the types and attribute values of Python AST nodes).

It can be pretty handy in some niche cases though.

(Trey Hunner - Python trainer)

But beyond that, it is hard to see the benefits outweighing the costs.

Verdict

Structural Pattern Matching should be for very narrow situational use only. Match case is not an important alternative for conditional logic in Python. In a narrow range of specialist cases it is probably the right tool for the job but it should not be thought of as Common Python.

Trey Hunner - Twitter ↩

Comprehensions are Common Python

2022-10-02T18:00:00+13:00

What has your experience of comprehensions been? Do you agree with this post? Do you disagree? Please comment below.

Overview

Comprehensions are well covered elsewhere so we'll only briefly introduce them below before making the specific points relevant to the When of Python. In summary, it will be argued that comprehensions are such an important part of Python we should treat them as Common Python but that we should only use this compressed syntax when it is relatively easy to understand.

Comprehensions

List Comprehensions

List comprehensions are a good starting point to explain the syntax and value of Python comprehensions. A common need in programming is to take a collection of items and return a modified or filtered collection. For example, maybe we want the square of every number in a list: [1, 2, 3] => [1, 4, 9].

We could create this new list using a for loop:

numbers = [1, 2, 3]
squares = []
for num in numbers:
    squares.append(num ** 2)

Or using map we could write:

def square(x):
    return x ** 2

numbers = [1, 2, 3]
squares = map(square, numbers)

In pseudocode what we want is collect into a list the square of every item in the numbers list. In Python we can write code where the syntax is very similar to the pseudocode using a comprehension:

## squares is a list of the square of every item in the numbers list
squares = [num ** 2 for num in numbers]

And if we want to only include numbers < 3 the syntax is still very similar to a simple pseudo code version of what is desired.

## squares is a list of the square of every item in the numbers list
## if the number < 3
squares = [num ** 2 for num in numbers if num < 3]

See also How to write a Python list comprehension in 3 simple steps for a useful animation.

Other Comprehensions

Python 3 introduced other comprehensions including dictionary and set comprehensions. For example:

names = [('Charles', 'George'), ('Camilla', 'Shand'), ('Charles', 'Dickens')]
unique_fnames = {fname for fname, _lname in names}

Comprehensions are Common Python

Comprehensions are Common Python (see Welcome to the When of Python). This point was actually controversial in a Python mailing list debate in 2018. The objection to treating comprehensions as Common Python was that developers working with Python often come from other languages which don't have comprehension constructs. It was suggested that little would be lost avoiding their usage but the benefit would be that Rust developers, for example, could work on Python code without having to learn an unfamiliar concept.

Against this position I think comprehensions are an elegant, readable way of expressing simple tasks into relatively simple code (once the approach has been learned). For a defence of comprehensions I refer to Raymond Hettinger's classic talk on Transforming Code into Beautiful, Idiomatic Python. Hettinger argues that we shouldn't break atoms of thought into subatomic particles. He advocates instead for concise, expressive one-liners where one logical line of code equals one sentence in English.

Another advocate of comprehensions¹ is Rodrigo Girão Serrão What learning APL taught me about Python. He suggests the reason comprehensions are so good is their readability and that the reason they're so readable is that they highlight the most important part of the code, the data transformation. So instead of being hidden away at the bottom-right of the code, like it is in a loop, the transformation is prominently displayed top-left.

Comprehensions are a beautiful part of Python and we shouldn't cripple the language to keep it compatible with as many other languages as possible.

Certainly, comprehensions are very widespread in Python code, so we should include them in Common Python - that is, those features we should all know so we can basically read each other's work.

Note - just because we consider comprehensions Common Python and not Situational Python doesn't mean we shouldn't be thoughtful about when to use them and when not to.

Incomprehensions

Being one-liners (even if stretched over several lines) comprehensions can attract the same "cleverness" which plagues a lot of code. The desire to fit complexity into a single line can overwhelm concerns with readability, maintainability, and avoiding bugs. As a general rule of thumb, if a comprehension stops being easy to read, use an alternative approach instead e.g. a for loop or map. Note - Walrus operators in comprehensions should generally be viewed as "tricky" and not worth the cost.

By his own admission, Rodrigo is partially obsessed with comprehensions and considers them "insanely useful" - he has even written a book with over 200 exercises on the topic! ↩

Classy Data with Dataclasses

2022-09-25T23:00:00+13:00

What has your experience of dataclasses been? Do you agree with this post? Do you disagree? Please comment below.

Introduction

Passing data around can easily become confusing. Dataclasses are a fantastic way of structuring and documenting our data and we should be using them a lot more (see Why You Should Use Data Classes in Python). Maybe we should stop teaching collections.namedtuple and typing.NamedTuple and focus on making dataclasses.dataclasses as idiomatic, familiar, and readable as possible. This article compares different approaches to creating data structures and concludes with examples of dataclasses used in conjunction with type hinting.

Passing Data Around With Confidence

Passing data around is reasonably easy in Python. The syntax for tuples, lists, dictionaries, and sets is clean and easy to learn. Passing data around without being confused or uncertain - that takes more care. For example, imagine we are passing around coordinate data with latitude, longitude, and a code for the type of coordinate system. We can do it in a positional way or with keywords. But which is easier to deal with? (35, 35, 27200) or {'lat': 35, 'lon': 35, 'srid': 27200}? Which approach is most likely to result in bugs? And which approach is going to cope easiest with extra items and reordered data? While there are cases where plain tuples are a good choice the moment readability is jeopardised we should use a data structure option with keywords. But which type of keyword data structure should we choose?

Dicts vs namedtuples vs NamedTuples vs Dataclasses

Dictionaries are very flexibile and mutable - which is their greatest strength and their greatest weakness. When you receive a dictionary you can never be sure which keywords it will be using or whether it has been tampered with. Perhaps someone added the 'latitude' keyword and the dictionary is now:

{'lat': 35, 'lon': 35, 'srid': 27200, 'latitude': 35}

Dictionaries also require extra boilerplate to construct.

Collections.namedtuple provides a good solution where a dot notation is possible, keywords are fixed, data can (more often) be trusted, and yet it is possible to populate them in a lightweight way if that makes most sense. For example:

from collections import namedtuple

Coord = namedtuple('Coord', 'lat, lon, srid')

## Lightweight usage
place_a = Coord(-36, 174, 4326)
place_b = Coord(-37, 175, 4326)

## More explicit usage (if so desired)
place_c = Coord(lat=-36, lon=175, srid=4326)

Better Handling of Defaults and Item Types

typing.NamedTuple offers an even better approach with a very readable syntax. E.g.

class Coord(NamedTuple):
    lat: float
    lon: float
    srid: int = 4326

If you are working with a typing.NamedTuple Coord in your code it is very easy to see just what you are dealing with and to reason about it. Merely hovering over a type hint or instantiation will be enough to understand what data you are receiving / passing without having to lose focus on the actual coding.

It is hard to overstate how valuable this in when passing around non-trivial data structures. It also proves its worth when changing the details of your data structure - it is very easy to introduce bugs when changes aren't propagated successfully throughout the code. Having a clearly defined (and discoverable) structure makes it easy to search for uses and make consistent changes with confidence.

Type Hinting and Data Types

Type Hinting

Type hinting is an important part of modern Python - whether used as glorified comments or for stricter static type checking purposes (see Get the Hint - Type Hinting is Common Python. Used judiciously it can make code much more readable.

For example, which is easier to understand - a function definition without type hinting:

def create_map(coords):
    ...

or a definition with hinting:

def create_map(coords: Sequence[Coord]) -> str:
    ...

In the case where hinting is used we can hover over Coord in our IDE and see precisely what we should expect including keywords to use and types.

Note - this is not an argument for type hinting everything all of the time. That is a separate debate. The point here is that type hinting, combined with a readable type of data structure, can make code much safer and more pleasant to work with.

Why Dataclasses?

This is a fair question - typing.NamedTuple is a very good solution - so why do we need dataclasses? In some ways it is a close call and initially I concluded that we didn't in most cases ¹. There are some good arguments for using named tuples in some cases as explained in namedtuple in a post-dataclasses world. But I've changed my mind more recently because of the need to constrict Python for all the reasons covered in the original When of Python talk. The unique benefit of dataclasses is that we can use them for all our keyword data structure needs - whether mutable or immutable. This is not true of named tuples. We can also focus our teaching / learning efforts on using dataclasses and using them well. We can forget about collection.namedtuple's approach to default arguments and its different ways of defining fields. We can forget about remembering the difference between namedtuples and NamedTuples. We can forget about the sometimes subtle differences between dataclasses.dataclass and typing.NamedTuple². In short, we can ensure there is One Obvious Way of creating keyword data structures.

Note - depending on how they're used³ it may not be too hard to swap out typing.NamedTuples for dataclasses.dataclasses:

## typing.NamedTuple
class Coord(NamedTuple):
    lat: float
    lon: float
    srid: int = 4326

## dataclasses.dataclass
@dataclass(frozen=True)
class Coord:
    lat: float
    lon: float
    srid: int = 4326

Bringing it Together

Type hinting with dataclasses makes it very easy to know what data functions expect and what data they'll produce in a way that tuples, lists, dictionaries, and sets do not.

To illustrate:

from dataclasses import dataclass
from typing import Sequence

WGS84 = 4326

@dataclass(frozen=True)
class Coord:
    lat: float
    lon: float
    srid: int = WGS84

...

## Hundreds of lines of code later, or in another module

coords = []
for latitude, longitude in data:
    ## When writing this code I know which keywords I have to play with
    ## I just hover over Coord and everything is very clear
    coords.append(Coord(lat=latitude, lon=longitude, srid=WGS84))

...

def make_map(Sequence[Coord]):
    for coord in coords:
        if coord.srid != WGS84:
            raise ValueError(
                f"Unexpected SRID - got {coord.srid} instead of {WGS84}")

Maybe we should stop teaching collections.namedtuple and typing.NamedTuple and focus on making dataclasses.dataclasses as idiomatic, familiar, and readable as possible. There are big benefits for the Python community in having One Obvious Way of creating keyword data structures. For all the reasons covered in the article, dataclasses are a classy way of working with data.

Python Named Tuples vs Data Classes ↩
For example, no special function is needed to unpack or iterate over named tuples ↩
Vanilla dataclasses can't be unpacked or iterated over without special functions so code relying on that will break without other modifications ↩

The When of Lambda

2022-09-22T08:00:00+12:00

What has your experience of lambda been? Do you agree with this post? Do you disagree? Please comment below.

Lambda Examples

lambda x: x**2

A lambda defines an anonymous, one-liner function which is usually very short. They are widely encountered in code and are idiomatic in at least two use cases:

1) Pandas e.g.

.apply(lambda row: row['w'] * row['h'])

2) Sort e.g.

sorted(users, key=lambda user: user['name'].lower())

When of Lambda

So when should we use them? And when shouldn't we? In other words, can we create a When of Python for lambda? Trey Hunner's article Overusing lambda expressions in Python is highly relevant and I won't repeat all the great content in there. His main concern seems to be about the nameless nature of lambdas. Trey Hunner regularly teaches Python and places a high weight on readability. He makes a strong case that lambdas are overused and concludes with the following criteria:

I’d say that using lambda expressions is acceptable only if your situation meets all four of these criteria:

The operation you’re doing is trivial: the function doesn’t deserve a name

Having a lambda expression makes your code more understandable than the function names you can think of

You’re pretty sure there’s not already a function that does what you’re looking for

Everyone on your team understands lambda expressions and you’ve all agreed to use them

I would agree with those principles but want to modify the fourth. Perhaps we would be better off accepting the two use cases described earlier (pandas and sorting) and concentrate on improving their use in that context.

Idiomatic Use Clarified

Pandas

Using Pandas requires understanding lambdas and being comfortable with them. But we should never make complex lambdas. The moment a lambda become at all difficult to understand we should make a proper named function and use that.

"If a function is important, it deserves a name" (Trey Hunner)

And we should at least hint at the meaning of variables we use. For example, if we are processing rows we could use lambda row: row['height'] * 100 instead of lambda x: x['height'] * 100. Even lambda r: r['height'] * 100 would be good, especially if r and c became idiomatic short-hands for row and column in such cases.

Sorting

There is an alternative available using itemgetter but, to be honest, it is easier to use lambda and lambda is arguably more readable for most Python coders. Hunner presents the following comparison:

# Without operator: accessing a key/index
rows_sorted_by_city = sorted(rows, key=lambda row: row['city'])

# With operator: accessing a key/index
from operator import itemgetter
rows_sorted_by_city = sorted(rows, key=itemgetter('city'))

but draws a different conclusion from myself on readability. I like the lambda better because it is arguably idiomatic.

Incidentally, imagine if Python had a different syntax for lambdas and we could write:

# Without operator: accessing a key/index
rows_sorted_by_city = sorted(rows, key=(row => row['city']))

I guess it comes down to a matter of taste but I like the => syntax. In the Pandas context it would be especially nice:

.apply(x => x**2)

## vs

.apply(lambda x: x**2)

Conclusion

Lambdas are a Python language feature we should provide clear guidance on. Used well they simplify code and are perfectly readable; used poorly, and code becames opaque and bug-prone. Although use should be restrained, an accommodation should be made for Pandas and sorting, albeit with caveats to ensure usage doesn't compromise readability.

The Walrus and the (Software) Carpenter

2022-09-18T10:00:00+12:00

What has your experience of the Walrus operator been? Do you agree with this post? Do you disagree? Please comment below.

The Walrus and the Carpenter - Lewis Carroll

...

The Walrus and the Carpenter
Were walking close at hand;
They wept like anything to see
Such quantities of sand:
If this were only cleared away,'
They said, it would be grand!'
...

Lewis Carroll - Through the Looking Glass

The Walrus

The Walrus operator is an assignment expression. Normally we assign values to names with the syntax name = value e.g. fname = 'Zac'. The Walrus operator allows us to perform assignment as an inline expression within another statement. For example, instead of:

fname = 'Zac'
print(fname)

we can save a line by writing:

print(f_name := 'Zac')

The Software Carpenter

"Software Carpenter" in this post means a skilled Software Engineer - someone who understands the craft of software development¹.

As an analogy I've built some very basic structures in my garage out of left-over pieces of timber.

They solve a practical need but would make a woodwork teacher cry. A skilled carpenter would build them much better.

In the software world, there is a similar distinction to be made between hacking together code and being a Software Carpenter.

A Software Carpenter will care about the following:

Readability and Maintainability

Apart from throwaway code we need to care about readability and maintainability. In addition to unit testing and type hinting the best way of preventing bugs is to make code easily intelligible. This is especially important in projects which run for a long time and involve lots of different coders. And if code is mission critical or "enterprise" then it is especially important that code is written with readability as a high priority.

Clever Coder not "Clever" Code

Writing the simplest, most readable code possible is not easy. But that is the goal for a Software Carpenter. It can be fun to make the most dense, cryptic code possible - see Code Golf. But we should never do this in production code i.e. code that matters. "Clever" code is a breeding ground for bugs.

Walrus Examples - Simple, Readable Code?

I recently asked for example code that demonstrated the value of the Walrus operator. The following was received:

if ( transaction_type := retrieve( transaction ).type ) == "balance":
    show_balance()
elif transaction_type == "deposit":
    add_balance()
elif transaction_type == "withdrawal":
    decrease_balance()
else:
    raise TransactionException( transaction_type...doesn't make sense )

But avoiding the Walrus operator only adds one line (as was acknowledged up-front by the contributor) - namely:

transaction_type = retrieve( transaction ).type

Using Python without the Walrus operator also simplifies the first condition clause. Instead of:

if ( transaction_type := retrieve( transaction ).type ) == "balance":

we have:

if transaction_type == "balance":

So not a very compelling use case there.

Another example was discussed in Walrus Hunting with StrEnum. In that case StrEnum seems a better solution. The person suggesting the code agreed but said the example helped the Walrus operator click for them. Fair enough, but the search for a genuinely useful use case continues.

Another developer suggested a good use case for the Walrus operator was²:

filtered_list = [res for item in items if (res := slow(item))]

which seemed more persuasive. The slow function is only called once per item. The non-Walrus, non-comprehension alternative is:

filtered_list = []
for item in items:
    res = slow(item)
    if res:
        filtered_list.append(res)

which I have mixed feelings about. It is a lot longer. But its meaning is so plain it is less likely to contain bugs.

In balance, perhaps, a Walrus operator is a good option in such a case. It is both elegant and efficient. This use case is arguably the most persuasive of those presented in Should You Be Using Python’s Walrus Operator? (Yes. And Here’s Why).

But is a single use case of uncertain value enough to recommend teaching the Walrus operator to beginners? PHP is a language which has blended assignment and evaluation with often mind-boggling results - it is arguably a warning against this sort of "cleverness". Even an article promoting use of the Walrus operator (Should You Be Using Python’s Walrus Operator? (Yes. And Here’s Why)) acknowledges that the Walrus operator can be abused and it provides a useful section on "Gotchas and Limitations". If we want to shrink Common Python then tightly restricting the use of the Walrus operator is probably a good idea. Maybe it should be in Situational Python with a recommendation to limit usage to comprehensions.

Current Verdict

Until some compelling use cases for the Walrus appear we should probably discourage its widespread use in a When of Python. Mixing assignment and evaluation is a risky exercise and it would be better to invest energy understanding other safer and more useful language features or patterns instead. The one exception might be the use of the operator to avoid calling functions twice in comprehensions. The risks of this usage might be diminished if it became an idiomatic option (much the same way we often use lambdas when sorting). But generally speaking it might be best if Software Carpenters minimise contact with the Walrus:

...

It seems a shame,' the Walrus said,
To play them such a trick,
After we've brought them out so far,
And made them trot so quick!'
...

Apologies if this is confusing to anyone. I am aware that the emphasis in Software Carpentry is on more basic skills but the Lewis Carroll reference was too good to pass up. ↩
The example has been altered to avoid using the keyword list as a variable name ↩

Walrus Hunting with StrEnum

2022-09-14T20:00:00+12:00

What has your experience of the Walrus operator been? How about Enum and IntEnum? Do you agree with this post? Do you disagree? Please comment below.

Walrus Operator - Finally a Useful Use Case?

The Walrus Operator

The Walrus operator is named because := looks like the eyes and tusks of a walrus of its side. I really like the name¹ so it is a shame the Walrus operator isn't a useful addition to Python. Instead, the Walrus operator seems to be a solution looking for a problem. In spite of all the blog posts and tweets illustrating how it works there is very little that tries to explain why it is better than alternative approaches (unless we assume opaque one-liners are better than readable two-liners ;-)).

As a quick reminder, the Walrus operator is an assignment expression. Normally we assign values to names with the syntax name = value e.g. fname = 'Zac'. The Walrus operator allows us to perform assignment as an inline expression within another statement. For example, instead of:

fname = 'Zac'
print(fname)

we can save a line by writing:

print(f_name := 'Zac')

This is not an obvious improvement - indeed it can be argued that the Walrus operator commonly makes Python less readable and increases the complexity of basic code for no payoff. But perhaps it has other higher-value use cases?

One suggestion concerns collections of constants.

Constants

We often use "constants" to avoid magic numbers. E.g.

PG_PORT = 5432

...

port=PG_PORT  ## more semantic than port=5432

We can also use them to identify strings that we wish to control as developer, sometimes in collections that cover available options. For example:

AUCKLAND = 'Auckland'
WELLINGTON = 'Wellingon'
CHRISTCHURCH = 'Christchurch'
DUNEDIN = 'Dunedin'

CITIES = [
    AUCKLAND,
    WELLINGTON,
    CHRISTCHURCH,
    DUNEDIN,
]

Collections of Constants and the Walrus Operator

Recently it has been suggested that the Walrus operator is a useful way of slimming down such constructs. Using the Walrus operator we can instead write:

CITIES = [
    AUCKLAND := 'Auckland',
    WELLINGTON := 'Wellington',
    CHRISTCHURCH := 'Christchurch',
    DUNEDIN := 'Dunedin',
]

While it is true that, in this case, the Walrus operator saves multiple lines and avoids duplication, there are better ways of achieving the same results. It would probably be better to use an enum to collect these strings together.

StrEnum - A Valuable Addition

StrEnum is part of Python 3.11² and it enables the following syntax:

from enum import StrEnum

class City(StrEnum):
    AUCKLAND = 'Auckland'
    WELLINGTON = 'Wellington'
    CHRISTCHURCH = 'Christchurch'
    DUNEDIN = 'Dunedin'

We can then refer to individual cities using dot notation e.g.

if city == City.AUCKLAND:
    pass

The following is True:

'Auckland' == City.AUCKLAND

We can also combine our StrEnum nicely with type hinting to clarify what sorts of strings we expect to receive in a function:

def arrange_travel(city: City):
    ...

This is arguably better than:

def arrange_travel(city: str):
    ...

because it provides clear guidance about the type of inputs expected.

StrEnums also play nicely in other contexts - for example, as dictionary keys. The following works:

city_popns = {
    City.AUCKLAND: 1_571_700,
    City.WELLINGTON: 202_700,
    City.CHRISTCHURCH: 369_000,
    City.DUNEDIN: 126_300,
}
## Both return 1_571_700
city_popns[City.AUCKLAND]
city_popns['Auckland']

Using StrEnum Like a Ruby Symbol

Ruby has a construct called a symbol. It's a string that is identified by a colon at the front of a variable name e.g. :fname. The usage is subtle but important - basically "if the textual content of the object is important, use a String. If the identity of the object is important, use a Symbol." Ruby Symbols vs. Strings. The main benefit is clarity of programmer intention i.e. readability.

In Ruby another benefit of favouring symbols over strings is that they are interned - that is, there is only one instance no matter how many times it is used. This is much less relevant to Python. Most strings will be automatically be interned anyway - see Optimization in Python — Interning.

So for Python the real benefits of StrEnums are in readability and, potentially, type enforcement through type hinting. When we use a StrEnum we know it has a special meaning and is part of a controlled collection. We know that supplying any random string might not work - the code is expecting something from the type of StrEnum indicated.

When the Type of StrEnum Matters

There are cases where the fact our StrEnum is not really a string matters. See class enum.StrEnum. The documentation suggests we can simply convert our StrEnum to a string type e.g. str(City.AUCKLAND). Another approach is to provide a property to the StrEnum. E.g.

class City(StrEnum):

    @property
    def str(self):  ## in case we ever need the actual string type
        return self.value

    AUCKLAND = 'Auckland'
    WELLINGTON = 'Wellington'
    CHRISTCHURCH = 'Christchurch'
    DUNEDIN = 'Dunedin'

Then we can write City.AUCKLAND.str instead if that is more to our taste.

StrEnums 1: Walrus Operator 0

In conclusion, the StrEnum is a new feature in Python we should start using a lot more. And if we do, it is even harder to think of cases where the Walrus operator is the best solution. Too harsh? Too kind? Add a comment.

Postscript - Testing StrEnum

Docker can be a convenient way of experimenting with unreleased (or old) versions of Python. The following lets you run scripts in your current working folder using the version of Python in the container.

docker run --rm -it --mount type=bind,source="$(pwd)",target=/src --workdir /src python:3.11-rc /bin/bash

I also like the way Ruby has the delightfully named spaceship operator <=> Ruby — The Spaceship Operator 101 ↩
It has been possible to use string Enums since 3.4 but it is even easier now that StrEnum has been added. ↩

Get the Hint - Type Hinting is Common Python

2022-09-13T06:00:00+12:00

What has your experience of type hinting been? Do you agree with this post? Do you disagree? Please comment below.

Type Hinting - A Big Deal for Python?

Some new features really define the future of a programming language. Long after the walrus operator has been forgotten, and structural pattern matching found its niche, type hinting will be shaping people's everyday Python - for better or worse. A lot depends on how we agree to use this language feature.

Currently, almost a third of active Python projects on Github are using type hinting for parameters (see The When of Python - KiwiPycon 2022). The sooner we figure out type hinting best practice the better.

To kick this blog post off I contacted Danny Adair - founder and former President of NZPUG - to get some tentative thoughts. I knew he had opinions on type hinting based on a conversation we had at the recent Kiwi PyCon. Knowing Danny was currently extra busy I asked for a quick reaction and here it is:

I think type hinting in Python can be useful, particularly if you (previously) enjoy(ed) the coziness of the harness that is static typing, and you're willing to sacrifice a duck for it.

I think there are some codebases that were screaming for it (like the one at Dropbox where Guido started pimping it) but I would argue against it as becoming a habit. Unfortunately "optional" stops being optional for all practical purposes once you touch type hinted code with your code.

"There should be one-- and preferably only one --obvious way to do it." - But don't worry it's optional...

(How's that for a rant)

Some important questions are raised about type hinting. Should it be restricted to a few enterprise codebases? Should it be a matter of personal taste - perhaps we should accept type hinting as an option for those who are happy to embrace the restrictions of a static typing approach but recommend against it generally? Can this be a matter of personal taste or is there a risk type hinting will draw everything in - potentially spoiling the dynamic, duck-typed flavour of Python which has been part of its success?

I'll invite Danny to add further comment when this post is published. But let's step back a bit now and refresh our memories on how type hinting works and different ways of using it.

Different Ways of Type Hinting

Type hinting allows us to indicate what types a function expects as arguments and what types it will return. E.g.

def get_greeting(name: str, age: int) -> str:

As the name makes clear, type hints are hints only but there are tools that enable type hinting to be checked and enforced.

Type hinting is still maturing in Python and more recent versions are less verbose and more readable e.g. int | float rather than typing.Union[int, float]

Initially I didn’t like type hinting. I suspected it was a costly and ritualistic safety behaviour rather than a way of writing better code. And people can certainly use type hinting like that. But I have changed my overall position on Python type hinting.

There are at least three ways to use type hinting in Python:

1) As Glorified Comments

We can use type hinting to improve readability and reduce confusion - basically to treat type hints like glorified comments. For example, what is the following function expecting for the date parameter?

def myfunc(date):

Is it OK if I supply date as a number (20220428) or must it be a string ("20220428") or maybe a datetime.date object? Type hinting can remove that confusion

def myfunc(date: int):

It is now much more straightforward to consume this function and to modify it with confidence. I strongly recommend using type hinting like this.

The following is very readable in my view:

def get_data(date: int, *, show_chart=False) -> pd.DataFrame:

Note: there is no need to add : bool when the meaning is obvious (unless wanting to use static checking as discussed below). It just increases the noise-to-signal ratio in the code.

def get_data(date: int, *, show_chart: bool=False) -> pd.DataFrame:

On a similar vein, if a parameter obviously expects an integer or a float I don’t add a type hint. For example type hinting for the parameters below reduces readability for negligible practical gain:

def create_coord(x: int | float, y: int | float) -> Coord:

Or, knowing that mypy considers int a subtype of float:

def create_coord(x: float, y: float) -> Coord

Instead

def create_coord(x, y) -> Coord:

is probably the right balance. To some extent it is a matter of personal taste.

David Beazley in his 2022 book "Python Distilled" takes a relaxed and pragmatic approach: "If the inputs and outputs of a function aren't clear from their names, they might be annotated with types" (p.23)

I hope people aren’t deterred from using basic type hinting to increase readability by the detail required to fully implement type hinting.

2) To Enable Static Checking

Another option is to use type hinting to enable static checks with the aim of preventing type-based bugs – which potentially makes sense when working on a complex code base worked on by multiple coders. I doubt we should do the same for ordinary scripts – the costs can be very high (see endless Stack Overflow questions on Type Hinting complexities and subtleties).

3) Ritual Self-Soothing

For some people I suspect type hinting is a ritual self-soothing behaviour which functions to spin out the stressful decision-making parts of programming. Obviously I am against this especially when it makes otherwise beautiful, concise Python code “noisy” and less readable.

Duck Typing

Time for a Duck Sacrifice?

Danny's comment on duck sacrifice may be puzzling for some. To start with, it has nothing to do with duck punching as per Monkey patching and Duck punching ;-). The issue is whether we lose or weaken the duck typing aspect of Python if we let type hinting spread too freely.

Python follows a Duck Typing philosophy – we look for matching behaviour not specific types. If it walks like a duck and quacks like a duck it’s a duck!

For example, we might not care whether we get a tuple or a list as long as we can reference the items by index. Returning to the Duck illustration, we don’t test for the DNA of a Duck (its type) we check for behaviours we’ll rely on e.g. can it quack?

There are pros and cons to every approach to typing but I like the way Python’s typing works: strong typing (1 != '1'); dynamic typing (defined at run-time); and duck typing (anything as long as it quacks).

Fortunately, no duck sacrifice is necessary if we mainly use type hinting as glorified comments and, in the context of more disciplined static type checking, rely on Protocol to allow behaviour-based (structural) hinting.

Structural Type Hinting using Protocol

If we were able to blend type hinting with duck typing we would get something where we could specify accepted types based on the behaviours they support.

Fortunately this is very easy in Python using Protocol. Below I contrast Nominal Type Hinting (based on the names of types) with Structural Type Hinting (based on internal details of types e.g. behaviours / methods)

Full Example in Code

from typing import Protocol

class BaseAttacker:
    def attack(self):
       raise NotImplementedError

class Soldier(BaseAttacker):
    def __init__(self, name):
        self.name = name
    def attack(self):
        print(f"Soldier {self.name} swings their sword!")

class Archer(BaseAttacker):
    def __init__(self, name):
        self.name = name
    def attack(self):
        print(f"Archer {self.name} fires their arrow!")

class Catapult:  ## note - not inheriting from BaseAttacker
    def __init__(self, name):
        self.name = name
    def attack(self):
        print(f"Catapult {self.name} hurls their fireball!")

## only accept instances of BaseAttacker (or its subclasses)
def all_attack_by_type(units: list[BaseAttacker]):
    for unit in units:
        unit.attack()

s1 = Soldier('Tim')
s2 = Soldier('Sal')
a1 = Archer('Cal')
c1 = Catapult('Mech')

all_attack_by_type([s1, s2, a1])

## will run but won't pass static check
## because c1 not an BaseAttacker instance
## (or the instance of a subclass)

## comment out next line if checking with mypy etc - will fail
all_attack_by_type([s1, s2, a1, c1])

class AttackerProtocol(Protocol):
    def attack(self) -> None:
        … ## idiomatic to use ellipsis

def all_attack_duck_typed(units: list[AttackerProtocol]):
    for unit in units:
        unit.attack()

## will run as before even though c1 included
## but will also pass a static check
all_attack_duck_typed([s1, s2, a1, c1])

Unexpected Conclusion

Type hinting is useful as glorified commenting and as such should be part of everyone's Everyday Python. In other words it should be part of Common Python. Type hinting is likely to be widely used in the code people encounter (see earlier comment on usage in Python projects on Github) so we all need to be comfortable reading simple type hinting.

Some enterprise and library codebases will benefit from a more strict and enforced form of type "hinting" and further advances in type hinting and checkers like mypy will improve the developer experience of using this feature.

The dynamic flavour of Python can be maintained by using type hinting in this way and by making use of Protocol. No duck sacrifice is necessary.

If we use type hinting to improve the readability of our Python code generally, and to increase the robustness of enterprise and library code specifically; and if we can avoid using type hinting to make Python less flexible and Just Another Static Language, then type hinting will be a great addition to the language.

Deprecate for else or else?

2022-09-05T21:00:00+12:00

The for else construct seems simple enough - the problem is that the correct interpretation of how it works is counterintuitive to a large proportion of Python coders. For example, which of the following is correct? It's not immediately obvious whether the else block runs if the break is hit, or if it is not hit.

for animal in animals:
    if animal == my_pet:
        break
else:
    print("No pet is found :-(")

for animal in animals:
    if animal == my_pet:
        break
else:
    print("Pet found :-)")

Some suggestions for better names than else have included nobreak and (less seriously) if_we_exited_the_block_without_encountering_a_break.

In practice, it's clearer to just remove the ambiguity for the reader and set a variable before breaking.

found_pet = False
for animal in animals:
    if animal == my_pet:
        found_pet = True
        break
if not found_pet:
    print('No pet found :(')

If Raymond Hettinger had a time machine he could tell Guido in the future no one will find else intuitive: Transforming Code into Beautiful, Idiomatic Python. Incidentally this is one of the classic Python presentations - watch it and bump the views over from 999,975 to a clean 1M ;-)

Guido says if he had a time machine he would not have included the feature at all: Python-ideas for/else syntax.

Maybe we should put for else in the Deprecate Python category? Why not - there are alternatives that are versatile, simple, and safe.

Fair comment? Overly harsh? What do you think?

Pickle in a Pickle

2022-09-04T23:00:00+12:00

Python pickle is something many of us have had a love-hate relationship with. We've loved it before using it and hated it afterwards ;-). Even the official documentation at pickle — Python object serialization seems a bit faint in its praise.

Maybe we should put pickle in the Deprecate Python category?

Fair comment? Overly harsh? What do you think?

Note - the following comments were transferred over from a more general blog post about features so all the pickle content can be together.

Python Feature Opinions

2022-09-03T11:00:00+12:00

Your Opinions Welcome

The When of Python is at heart a community project. We need people to share their opinions on different Python language features. If you have opinions on any language features we'd love to hear from you.

E.g. here is what a comment on lambda might look like:

Language feature: lambda

Common Python / Situational Python / Deprecated Python: Situational (see Welcome to the When of Python for definitions)

Opinion: Should only be used in sorting e.g. .sort(key=lambda ...) and, carefully, in Pandas e.g. .apply(lambda ...)

Any Interesting References (optional): Overusing lambda expressions in Python - Trey Hunner

Bio (optional): up to you - Anon is OK

This format is only a suggestion.

But please fire something off to us, either in the comments, or if you prefer, privately (whenofpython@p-s.co.nz).

We know people have opinions. Is type hinting a brilliant new feature; a blight on Python; or something else? Was the walrus operator worth the grief? Is structural pattern matching as good as it initially seems? Should dataclasses replace collections.namedtuple and typing.NamedTuple? Etc. You get the idea :-)

Feature Creep vs Easy On-Ramp

2022-09-02T09:00:00+12:00

Learning Python

Python has a deep heritage as a learning language. The creator of Python, Guido van Rossum, worked for several years on ABC which was intended for teaching and prototyping. Python was born out of that context (see History and Philosophy of Python: Easy as ABC) and learnability is baked into its DNA.

As we were developing the ideas that would lead to the When of Python, we started thinking about whether Python was still as easy to learn as it used to be, given there is now often more than "one obvious way" to do things. We also wondered whether having a variety options for how to approach a task was necessarily a bad thing; it provides greater flexibility to the user after all.

This got me thinking about my experience with the X-Wing Miniatures Game...

The X-Wing Miniatures Game

A few years ago I started playing the X-Wing Miniatures Game. Compared to similar games, the simplicity of the game was a major drawcard, because it meant that I could easily teach my friends how to play! Instead of having a hardback rulebook with hundreds of pages, all you needed to get going was a lightweight rules pamphlet and a handful of cards and tokens:

Like many games, over time they released new expansions to provide new possibilities for your squadron. Not only did each expansion add a new ship to the game, but they also added new pilot abilities, ship upgrades, and game mechanics:

Each new expansion was a welcome addition for veteran players, as it helped keep the game fresh and exciting. However, after a few years of expansions, the sheer volume of added rules had created a steep learning curve that made the game unapproachable to beginners:

In the end, the game designers had to release a second edition to drastically simplify the game because it had simply become too complex:

"...it got more complicated, making it less easy to jump into the proverbial cockpit."
Fantasy Flight Games Hits Lightspeed with X-Wing Second Edition

Releasing a second edition was no small endeavour: every player had to replace all of their existing cards and tokens with updated versions.

Learnability

Like X-Wing, learnability has been a big part of Python's success. Python recently surpassed Java as the top learning language in universities:

That success has been attributed to its simplicity:

Python possesses a mix of qualities that makes it a good candidate for universities. It has a simpler syntax than Java or C++, allowing novices to start writing programs almost immediately.
Python bumps off Java as top learning language - Eight out of the top 10 universities now use Python to introduce programming - Joab Jackson

And just last year, Python became the most popular language on the TIOBE index:

Again, simplicity is arguably a major factor in that success:

"Python, which started as a simple scripting language, as an alternative to Perl, has become mature. Its ease of learning, its huge amount of libraries and its widespread use in all kinds of domains, has made it the most popular programming language of today," said TIOBE CEO Paul Jansen.
Python ends C and Java’s 20-year reign atop the TIOBE index - Brandon Vigliarolo

Threats to Python's Learnability

But how easy is it really to learn modern Python? In order to confidently read code, we have to be familar with all the language features that we're likely to come across, Consider how much more needs to be at least mentioned in a Python crash-course today compared to just 10 years ago. We now have type hinting, f-strings, the wonderfully-named walrus operator (:=), positional-only parameters, and structural pattern matching (match). And that's just what's been added in the last 10 years!

With so much more to learn, Python risks becoming unapproachable to beginners, especially those who aren't full-time software engineers, such as scientists, school teachers, and data analysts.

By alienating those users that have made Python so widespread, Python's language creep could ultimately threaten its popularity.

The When of Python

Guidance on when to use Python features, and when not to, could radically simplify the task of learning (and teaching) Python. The When of Python could achieve this by eventually capturing a community consensus on the features to associate with Common Python, Situational Python, and Deprecated Python. In short, the When of Python could help Python remain true to its teaching language roots, which have been such a crucial ingredient in its huge success to date, and hopefully will remain so!

The Who of Python

2022-09-01T22:02:00+12:00

Python is used by very different groups of people and how we decide which features are Common Python and which are Situational Python must take that into account. If something is common to one group only e.g. data scientists, or enterpise web application developers, it should be considered Situational Python even if the use case is very common for that group.

Nick Coghlan loosely and tentatively divided Python use cases up in a 2017 blog post (Considering Python's Target Audience Nick Coghlan 2017-10-09):

Education
Personal automation & hobby projects
Organisational process automation
Set-and-forget infrastructure
Continuously upgraded infrastructure
Intermittently upgraded standard operating environments
Ephemeral software
Regular use applications
Shared abstraction layers

He also suggested that largest split in Python's user base is between Python as a scripting language and Python as an application development language.

Another useful split would be between full-time developers and people who are astronomers, biologists etc first, and developers second.

However we divide things up the main point is that there is no such thing as an Everyday Python that is the same for everyone. But having said this, there is still a need for a Common Python so we can all basically read each other's work.

One Obvious Way - An Ongoing Challenge

2022-09-01T22:01:00+12:00

From the Zen of Python to the When of Python

There should be one-- and preferably only one --obvious way to do it. Although that way may not be obvious at first unless you're Dutch.

Python prides itself on having one obvious way to do things but it is more of an ideal than a reality sometimes. Nonetheless we should aim in that direction to increase code readability and reduce the barriers to learning and mastering Python.

The When of Python will hopefully make it easier to identify the preferred approaches to general tasks, and the preferred alternatives for specific situations.

True 11 years ago, true today

It was spooky reading a post of Nick Coghlan's from eleven years ago. It harmonises very well with a key concern of the When of Python - namely, making Python easier for learners. And not just people starting off as full-time developers. Python also needs to work for people who are astronomers, biologists etc first, and developers second. Anyway, here is what Nick had to say:

What do you mean by "cognitive burden"?

Even without considering the near-term cost of changes, every addition to the language (and even the standard library) imposes a potential burden on anyone learning the language in the future. You can't just say, "Oh, I won't worry about learning that feature" if the code base you've been asked to maintain uses it, or if it is offered as an answer to a query posted on python-list or Stack Overflow or the like. The principle of "There Should Be One - and preferably only one - Obvious Way To Do It" is aimed squarely at reducing the cognitive load on people trying to learn the language. Quite clearly, the "only one" aspect is an ideal rather than a practical reality (two kinds of string formatting and three argument parsing libraries in the standard library all say "Hi!"), but in such cases we do try to indicate that the most recently added (and hopefully least quirky) approach is the preferred way to do it.

(Musings on the culture of python-dev Nick Coghlan 2011-04-21)

Looking back from 2022, we've managed to simplify argument parsing. As far as string formatting goes, though, we now have four approaches! Oh well - two steps forward, one step back. Perhaps developing a When of Python will help. As we said in our Kiwi PyCon presentation, newer is not always better (or worse for that matter) so more explicit guidance is probably helpful.

Welcome to the When of Python

2022-09-01T22:00:00+12:00

The When of Python is a fledgling community initiative. The goal is to effectively shrink Python so it fits our brains by providing guidance on when we should use particular language features (and when we should not).

We start with Common Python - the features we should all know so we can basically read each other's work. Clarity on this will be especially useful for teachers and anyone learning Python.

Then there is Situational Python - the Python features that are useful but not for everyone or not all the time. It depends on the situation, for example:

Web development might need very different features than scientific Python, e.g. asyncio
Advanced library code might need advanced features, e.g. low-level threading / multi-processing

For each feature we try to provide guidance on what the situations are where it is the preferred One Obvious Way. The situations might be defined by domain (e.g. web application development vs data science) or type of code (e.g. scripting vs code with enterprise responsibilities).

Finally there is Deprecated Python - the features we should avoid using. And maybe we should go even further than that - perhaps working through existing code and expunging some features in favour of better alternatives. Soon(ish) we are planning to release some tools to make that easier.

Initial Resources

The When of Python presentation as delivered at Kiwi Pycon 2022, and an addendum of post-conference refinements:

Slides

Original presentation:

The When of Python

The Lean Python Blog - Python

Assert Assertions

Introduction

(Simplified) Guidance

Warning

Conclusion

String Theory

Concatenation, Gluing, and Interpolation

When Not to Use f-string Interpolation

f-string Tips (Format Strings)

Zero-Padding

Commas in Numbers

Percentages

Dates

Dataclasses Considered Sweet

Dataclasses Considered Sweet

Overview

Dataclasses as Foundational Data Structures

Dataclasses as Syntactic Sugar

Sweet Enough?

The Power of Sane Defaults and Python Packaging

The When of Python and Sane Defaults

Annointing a Sane Default for Python Packaging

Simple Async

Asynchronous Programming

Alternatives

Reach for Pool Executors First

Simple Pool Executor Recipes

The Challenge of Simplicity

The Ongoing Spread of Python

Everything Needs a Simple Interface

There Will be More Python

Deprecation Appreciation

Python as a Designed Language

Reality

Deprecation

Deprecated Python

Restrict Struct?

Introduction

Some Arguments

We need an alternative to namedtuple and dataclasses

We don't need methods on our data structures

Struct has an elegant syntax

Optional typing

Performance

Misc

Conclusion

Match Case - A Simple Switch Statement for Python?

Simple Switch Statement at Last?

Python Pattern Matching Problems

New Mini Language

Similarity to Object Instantiation Misleading

Similarity to isinstance Misleading

Use Cases (in Python)

Verdict

Comprehensions are Common Python

Overview

Comprehensions

List Comprehensions

Other Comprehensions

Comprehensions are Common Python

Incomprehensions

Classy Data with Dataclasses

Introduction

Passing Data Around With Confidence

Dicts vs namedtuples vs NamedTuples vs Dataclasses

Better Handling of Defaults and Item Types

Type Hinting and Data Types

Type Hinting

Why Dataclasses?

Bringing it Together

The When of Lambda

Lambda Examples

When of Lambda

Idiomatic Use Clarified

Pandas

Sorting

Conclusion

The Walrus and the (Software) Carpenter

The Walrus and the Carpenter - Lewis Carroll

Similarity to `isinstance` Misleading