What are some features in Python that you discovered many years later?
For me, it was the the option to use & to find intersections of two sets, | operator to unionize two sets, as well as ^ for symmetric difference
For me, it was the the option to use & to find intersections of two sets, | operator to unionize two sets, as well as ^ for symmetric difference
python -m http.server
python -m json.tool
You've got to be kidding. I've been using jsonlint.com all these years.
For me it was collections.Counter. Spent years writing manual frequency-counting loops with defaultdict(int) before realizing Counter just does it and also gives you .most_common() for free. Felt pretty dumb when I found it.
Also itertools.chain.from_iterable for flattening nested iterables without writing a list comprehension that looks like a puzzle.
Just wanna throw it out there that python 3.15 was just released and has support for flattening using unpacking in comprehensions:
lists = [[1, 2], [3, 4], [5]]
[*L for L in lists] # equivalent to [x for L in lists for x in L]
sets = [{1, 2}, {2, 3}, {3, 4}]
{*s for s in sets} # equivalent to {x for s in sets for x in s}
dicts = [{'a': 1}, {'b': 2}, {'a': 3}]
{**d for d in dicts} # equivalent to {k: v for d in dicts for k,v in d.items()}
(examples from https://docs.python.org/3/whatsnew/3.15.html)
Whoa that's very nice, thanks for the tip! I always mess this up when I try to do this manually
I've used `flattened = sum(nested_list, [])` , it's funny how it works! Sum has a second argument for default starting value to add the values into. It uses the fact that lists have `__add__` or `__iadd__` operand defined as list.extend and will put each nested element into that empty initial list!
I would say that this isn't really a good way to do it, as it results in O(n2) complexity. Ie when you flatten [[1], [2], [3], [4], ...], it'll do [] + [1], allocating a new list and copying all items, then do [1] + [2], allocating a new list and copying all items, then do [1,2] + [3], allocating a new list, and so on, so you're copying 1+2+3+4+.... elements into all these temporary lists.
Using something like list(itertools.chain.from_iterable(nested_list) is much faster, as it'll be adding to the same list, getting O(1) amortised appending, Eg. here's some timings to demonstrate:
l = [[0]] * 1000
%timeit sum(l, [])
815 μs ± 6.8 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
%timeit list(itertools.chain.from_iterable(l))
28.9 μs ± 1.21 μs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
So nearly 30x faster for a 1000 item list. But scale that up to 100,000 items, and chain scales linearly (ie. just 100x slower), taking 2.84ms, while the sum version takes 7.68 seconds, so if the list can potentially be large, this can matter a lot.
Plus, using gimmicky syntax which, by some
serendipitous side effect, yields the correct answer, is a very bad practice in any collaborative code base. Good luck to the person trying to make sense of the code.
That’s so interesting lol, nice find!
Same, not only that, Counter is much faster than other methods
defaultdict.
defaultdict is a very bad idea. You end up with something that looks like a dict, typing-wise is a dict, but never throws a KeyError. One day it gets passed through your system and ends up somewhere it isn't expected.
I've seen this cause intermittent issues in two very different codebases.
Use dict.setdefault instead.
It's a great idea when you want a defaultdict. e.g. If you want to build indexes lists.
buckets = default.dict(list)
buckets["Monday"].append(12)
Or you want to be able to do:
played["hearts"]["queen"] = True
without having to check if "hearts" exists.
If passing a defaultdict through your system causes an issue you don't understand your system.
The problem is played["bug"] returns {} rather than raising a key error (and also increases the dicts len), which can cause weird errors at a distance.
The general advice is to use a defaultdict to construct your object and then copy into a normal dict to stop this happening.
It's a bad design, dict.setdefault would be better than creates a new default value whether you use it or not. As always, rust does it better
map.entry(key).or_insert_with(Vec::new).push(value)
okay ... "general" advice from whom??
If you want to use rust, use rust.
I don't understand. Isn't setdefault doing the same thing.
Just dict it at the end.
It's not the same thing as a defaultdict, as you need to explicitly call setdefault to set the value, but regular lookups will raise a KeyError if the item doesn't exist.
Use defaultdict if you don't care that empty lookups create new items. Use setdefault if you do.
I use a defaultdict to build it, then convert it to a regular dict before passing it along to the rest of the program, e.g. return dict(result)
Print the variable name itself when debugging ``` foo:str = “bar”
print(f”{foo=}”)
You can do what?!?
That was my reaction lol
It’s called self documenting expressions
Mariatta has a whole series of fabulous talks on f-strings I highly recommend
It even works with full expressions and maintains the white space you include within the {}
``` a = 4 b = 7
print(f"{b % a = }")
print(f"{(lambda x, y: x * y - x / y)(b, a) = }")
```
I felt like a wizard when I saw that this was doable in C++, interesting to see the python method.
I was so happy when this came out
Can you do this in python 2.5 (defense work)?
Not exactly a Python feature but I slept on Cython for a long time. Once I got into C I realized how easy it is to write extensions for Python using Cython.
Cython is a dead library walking.
The issue is simple - to get the really big speedups, you have to stop writing in Python, and use their weird Cython language. Now you can't ever run this code in Python again.
And you still don't get the same speedup as simply writing in Rust, C++ or C.
Now, this made a lot of sense in the past, because changing Python to Cython isn't too bad.
But now, you just prompt your coding assistant to port your Python library to Rust, and provide Python bindings at the same time, have a cup of coffee and it's done. Cython is simply an impediment.
You still need a rust compiler tool chain in your CI. With Cython you can just do it
I still need a C complier :) Rust is easy to install on Windows.
That with re.Match objects, they support __getitem__() so you can index into them instead of having to call .group() . So you can do match[3] instead of match.group(3); match["foo"] instead of match.group("foo"), etc. This was added in Py 3.6 and I slept on this for over half a decade.
I started with Python 2.6, over twenty years ago.
I read these sorts of threads to brush up, but in the last few years, I've really been seeing nothing surprising...
Until this. How did I not know? It's a conspiracy!!!
The worst is that it's completely logical and Pythonic. Had I ever thought about it, I'd have guessed that since this was the logical way to do it, it might work that way, and would have tried it out.
Bah! (Thanks!)
I think for me, I've looked over the re documentation about a thousand times now, but when I see Match.__getitem__(g) listed in the docs, my eyes skimmed over it without really understanding what it was trying to tell me. Probably it would help if match.group() was soft-deprecated and/or a note was there at the top informing people of Match.__getitem__(g) instead.
Same, started on 2.4 and I still have to dig up the docs to remember how to get match groups. Damnit!
That you can query dicts with tuple keys like this: d["a", 1], instead of d[("a", 1)].
This works most every place. The parentheses for a tuple are a disambiguating syntax in some places, and in others you don't need them. x = 'a', 1 sets x to a 2-tuple. The only places you need parentheses are in places that take commas, like in function argument lists or in list / dict / set literals for example.
The way I think about is that it's the comma that defines the tuple, not the parentheses. The parentheses just force the order of operation so all the items inside get evaluated as a tuple object before it's used by the rest of the expression or statement. This also explains why a trailing comma is required for 1 item tuples.
2-tuple
You mean a twople?
Whatt is this for when you have tuples as dict keys? are tuples really hashable? i guess so if they're immutable but... why
It's pretty handy when you want a sparse array. Eg. you're representing objects with a position, where you can just do d[x,y] = val to store live values, but 99%+ of the positions are empty. Using a dict instead of, say, a numpy array let's you have conceptually infinite multidimensional arrays that don't require space to store empty values, but still give you a similar interface.
You can do a simple string template like this:
t = "Hello {a}, my name is {b}"
d = {'a':'all', 'b':'Bob'}
print(t.format(**d))
You get:
Hello all, my name is Bob
Probably saw it a long time ago, then forgot about it. Would sure have been useful "a couple of times".
Although this is not that viable anymore (that is good though).
What do you mean?
Walrus operators and match/case really makes life a lot more convenient
Can you show an example of walrus?
Jokes aside, walrus operators are just a way to assign a value to a variable inside of a statement. Kinda like this:
if not (fpath:=Path(data_path).exists()):
fpath.mkdir()
I could have written it as:
fpath = Path(data_path)
if not fpath.exists():
fpath.mkdir()
And it would have worked exactly the same. I prefer wrapping them in parentheses so it's easier to read but they're not required.
I'd also like to say that it's little more than a nice shortcut but I'm certain someone much smarter than me will come along and tell me how stupid that statement is haha.
Sorry to be That Person but I think you meant:
if not (fpath:=Path(data_path)).exists():
...
The place where I most want a walrus is regexp matching.
if m := re.match(r'thing-(\w+)-also-(\d+)', s):
print(f'got {m[1]} for {m[2]}')
elif m := re.match(r'...', s):
...
I also find them really helpful in comprehensions
tags = {tag for t in raw_tags if (tag := t.upper().strip())}
Took me ages to realize you can chain those too, like a & b & c for a triple intersection
set intersection in math is associative and commutative so I would expect that to work anyway.
For me it was pathlib. Used os.path for years before realizing how much cleaner it is.
Also the fact that you can do dict1 | dict2 to merge dictionaries.
Yes! Then the ease you can check and create the path if it doesn’t exist.
pretty printing, using rich library.
import print from rich
Makes terminal output colorful and easy to work with. Specially with dictionaries and JSON payloads.
That you can give a tuple to starswith and endswith:
"string".startswith(('a', 'b', 'c'))
I've also reread contextlib documentation recently and it has quite some nice context managers: .closing() to auto call .close() on an object, .suppress() to ignore an exception, .redirect_stdout/redirect_stderr(new_target) and .chdir() are pretty nice and self explanatory. ExitStack is a bit weirder but seems pretty useful in some situation with a lot of files like objects https://docs.python.org/3/library/contextlib.html#contextlib.ExitStack. It seems that there is a push to replace some simple exception patterns with contextlib now.
There are also the importhooks system which allows you to run specific code when someone imports your module, it's very advance and mostly for tool building but it's nice to know it exists also.
Same for descriptors, it's what ORMs uses to bind attributes to database columns for a table represented by a class, it's an object on which you explicitly control how it's accessed and modified.
My code is littered with @contextmanager def chdir(...): ... because I keep forgetting that contextlib already has one!
suppress() is also something I keep forgetting about.
python -m json.tool
Just python -m json is enough since Python 3.14.
Pretty good when you're on some remote server and don't have jq available.
Lambda functions versus one-line defs
Use sparingly!
the only place I have used lambda functions is to define the sorting key in the sort function
I eventually stopped, after discovering operator.attrgetter/itemgetter/methodcaller.
I used to like sorted(list_of_tuples, key=lambda (a, b, c, d): b) back in Python 2 times, but Python 3 nerfed tuple unpacking, so I started using custom namedtuples/dataclasses instead of ad-hoc tuples.
Speaking of which, a feature of those that is often missed is that you can give multiple attributes/indexes and it'll give back a tuple of the items, so you can also do stuff like:
sorted(people, key=operator.attrgetter("lastname", "firstname"))
To sort by a compound key.
You wouldn't need to if python adopted a sensible notation for anonymous functions, like arrow notation, but that's just my gripe.
that we can do this
if 5
I'm still vaguely pissed off that this works, just because I don't understand how it's legal.
Assassin's creed voice Nothing is true. Everything is permitted.
I'm now angry at other languages for not permitting this.
TBH, I'm not sure it was really a great idea. It's cute, and occasionally handy, but I've also seen it cause confusion, because it works for all operators. I've seen newbies get confused by stuff like writing 1 < 2 == True and the like.
It's cute and all, but it's such a departure from most languages that I'm not sure how to feel about it. Breaking it up signals intent better, imho. Yes, if you're experienced with Python, this will not confuse you... But this still packs too much implicit knowledge of operator binding/precedence, and there's too many to watch out for. It doesn't cost much to write (5 < x) and (x < 10) instead
I mean it is what you would naively expect this to do with a math backgorund or no programming background. And iti s also what you want this to do 90%+ that you write it. Just because other languages have a shortcoming here doesnt mean you should follow it.
Three that come to mind:
`str.removeprefix()` / `str.removesuffix()` — for years I wrote slice
expressions like `s[3:]` with a comment explaining why 3, and every code
review turned into character counting.
`contextlib.suppress` — reads so much cleaner than try/except/pass when
you genuinely want to ignore one specific exception and nothing else.
`enumerate(seq, start=1)` — I used to keep a counter variable outside
the loop like it was C. The start parameter has been there forever.
You can make a general object with x = type("", (), {}) and just put stuff on it as needed, like x.foo = 3
I don't recall why this was useful, but I recall using it a lot earlier in my career, now it's probably a crime
Edit: and I don't use it anymore, relax, I'm not a monster
Btw. there’s SimpleNamespace for exactly that purpose.
Woah, dont do that.
There are occasional uses. PyTorch does this in a bunch of places, for example, and that's a very high-quality code base.
I have a huge number of old projects, and I vaguely remembered that I used it in exactly one, and a bit of grepping later, I found it in my most popular project:
https://github.com/rec/safer/blob/8f36fa367028a970e117d6b6012c54f6592e9ad2/safer/__init__.py#L729
What's going on here is that depending on whether you're windows or not, binary or not, and one other choice I forget, you end up with one of 8 possible classes for writing a file, and you can't even see the windows ones on a non-windows machine.
And then I need to override methods depending on your write mode for the file.
The result is an instance of a class that looks like the class you expect when you do open and it has the right type, but in which I have subverted many of the methods.
So I've used this feature once in twenty years. Very specialized.
EDIT: Downvoting without commenting is a poor choice.
Do that when appropriate.
Like all Python features, it's a tool. Whether it's appropriate depends on the context.
If you want arbitrary key-value pairs, you should just use a dictionary. The only reason to do this is if you really want dot notation over indexing, but that isn't a good enough reason to use this outside of personal or one-off script or projects.
Why not?
How could you?? There are children that read this sub!
But, kinda neat.
You rarely need, it, but when you do, nothing else will do.
If you use it a lot, it's probably a mistake.
Most of the time I see it, it's just a mistake.
Sometimes it could still be made more reasonable by using a local class:
def make_stringed(str_method: Callable[[Any], str]) -> type:
class Stringed(BaseClass):
__str__ = str_method
return Stringed
But the above example would likely be better using type, because you can set the name:
def make_stringed(str_method: Callable[[Any], str], name: str) -> type:
return type(name, (BaseClass,), {'__str__': str_method})
Cool. I always used x = object() / setattr
type("",(),{"foo":3})
I think I've seen this pattern suggested for implementing singletons. Definitely niche
I've used it during tool building to automatically generates classes on the fly based on a data structures (a syntax tree) for missing nodes, but that really pretty niche (and I've ended up dropping it)
The nonlocal keyword
I get the rationale for this but I’ve never used it. Typically it feels more Pythonic to use classes with instance vars than functions within functions.
Reverse avec Liste[::-1]
list(reversed(liste)) (or just reversed(liste) if you just need the iterator) is typically nicer to read. I would only do the above on the rare occasion I needed to reverse a string instead of a list.
list.reverse in-place will be better than both. Reversed make sense if you need to do additional stuff with the iterator. Reverse slicing is both unreadable and barely more efficient than list(reversed(list))
I tend to prefer the non-inplace versions rather than the in-place ones. This is where I pull out these quotes from Raymond Hettinger, core Python dev:
https://twitter.com/raymondh/status/1055478808793546752?s=20
https://twitter.com/raymondh/status/1055508781994594305?s=20
> Today's super-easy #python student question.
Q. What are the best practices for modifying a list while looping over it?
A. Don't.
Seriously, just make a new list and avoid hard-to-read code with hard-to-find bugs.
I did modify lists in-place when was I young, foolish, and had spare time for debugging ;-)
At some point, programmers eventually learn that immutability has its advantages and that building new structures is typically more straight-forward that modifying them in-place.
Well yea modifying a list while iterating over it should not even be possible in the first place and is a huge code smell IMO.
But here it's a single call, not the same.
I would also argue that if you care about immutability, then you should use a tuple instead of a list, which is much more efficient.
The only downside would be typing it, tuple[T,...] is more annoying than list[T], but this is also a non-factor because you can use Sequence[T] (which is what you should be using anyway regardless of the underlying data structure if you truly want to enforce immutability.
Constructing new structures each time is very inefficient if you don't need it. I prefer the paradigm: iterators chains, collect once as late as possible, then mutate once allocated, unless I find myself looping again.
I only recently discovered dictionaries were now guaranteed to be ordered. This has apparently been a thing since 3.7....
I was waiting for the 3.7 release specifically for this, and I was so happy when it came out!
On a related note, if you ever need an ordered set, just just a dict where the values are all None, and call set(d.keys()) when you're done adding to it and ready to read.
Won't the set constructor discard your carefully prepared order?
Yup, looks like you're absolutely right. Good catch!
Realizing I could use `__init_subclass__` to build a registry of base class implementations was a game changer.
There's also a subclasses iterator for every class, but IIRC it's a little fragile for registry construction. Like it won't return subsubclasses, IIRC
Yeah that's the easy way but gets kinda sketchy if the subclasses aren't all in the same file
Oh why does same file matter or not? Guess there's import order dependence
In any case, whenever I've looked into this I decided anything other than explicit registration is too brittle
This is one AI taught me.
its been awhile but it was after a while as well. But md5 function built in
Not so much a feature, but a pothole (kind of). Your average native C Python function (builtins like print(), etc) don't fully support the method descriptor protocol, so they cannot be bound methods. Even when assigned as class-level members, they'll always be static via instance-level access.
I just discovered * for unpacking.
I was utterly perplexed, catastrophically chagrined, and wonderfully delighted.
I learned about `for ... else` surprisingly late. The `else` block runs if the loop finishes without a `break`, including when there are no items. It’s handy for searches: break when you find a match; put the “not found” case in `else`.
That you can do `x = a or b` to assign to a or b if a is none.
you can do if (logic): function()
For me its object persistence with shelve. Say you have a dictionary you want to persist temporarily
You can do so
with shelve.open('spam') as db:
db['eggs'] = 'eggs'
Other times I create json or csv if I want to read or analyse them.
py_compile
When would you need to manually do it? Python does it for you automatically
use it for checking errors
List unpacking using *
l = [1,2,3]
l2 = [4,5,6]
final = [*l, *l2]
print(final) # [1,2,3,4,5,6]
Please don’t use “I” as a variable.. probably the worst choice..
You are right. It was just for demo purposes.
Type hints, generic typing, TypeVars, and (not a feature of python) mypy are all awesome, because you can basically turn python into typescript!
Python was my first language and I wrote it almost exclusively for 15yr. Thought I loved dynamic typing, until I wrote my first project in typescript... realized it has the same great readibility as python, but is faster, more versatile for the web imo, and static typing just makes everything less footgun-y
Seeing what set operators ('|', '&', '^', '-') do through visualization helps understanding: https://www.reddit.com/r/PythonLearning/comments/1tdgdt8/common_%F0%9D%98%80%F0%9D%97%B2%F0%9D%98%81_operations_in_python/
I hate that python decided to go this route with operator overloading instead of clearly named methods a the standard.
Except a few bytecode optimizations who barely make a difference there's 0 advantage for it
You have both silly:
a = b | c
a = b.union(c)
a = b & c
a = b.intersection(c)
# etc
You decide.
I know.
That's why I talked about "the standard". Python own documentation only use operators overloads
List and tuple concatenation are + and +=, for example, without a method alternative.
And even for sets, the standard as operators means that all code outside of my control will use this
I assume you are familiar with:
a = [1]
a += [2]
a.extend([3])
but there is none for a = extend2([4], [5]) or for tuple if you don't count things like a = [4].__add__([5]).
But operators are great, they are shorter and more readable. Hate all you want. Send an official complaint if you really need to.
Extend is not the same internal Cpython implementation as imul , it will be vastly more efficient to call the dunder if you know the Iterable is a list.
I struggle to see how obscure signs are more readable than plain english but to each their own.
The "shorter" argument makes even less sense for me.
LSP autocomplete make it in fact faster to type a method call than the operator syntax, for which they can't help much, and having 2 LOC instead of 1 is not something worth caring about.
Python is decades old at this point, complaining on python.org about this would be both useless and a bit dumb, especially since it's a subjective thing to argue about.
I wrote my library instead to never have to worry about it anymore in my own code, instead.
But as I said, all external code will still be written like this unfortunately
If you change your mind and do send a complaint, maybe also suggest changing:
a = 10
a += 1
a = a + 2
to:
a = 10
a.add_update(1)
a = a.add(b)
to get rid of those obscure signs.
Except here it make sense, because everyone not illiterate or newborn on earth know what it 1 + 2 means
My point is: use math signs for maths.
The fact that the add sign means totally different things for a numpy array and a list is a prime example of my complaint.
My main grip would be the C api. Great, to call set methods I need to call... PyNumber prefixed functions! Wtf?
Maybe just get literate:
Similar meaning for bit operations, common in most languages, yes there are differences.
Language is a creative use of syntax to begin with, compromises everywhere.
What is, and what is not, math.
I'm currently in uni and had a whole course on probability with lots of set related stuff. My knowledge on that isn't relevant. Words are still words. Operators are only meaningful in their context. I never need context with words: they ARE the context. Anyways, let's agree to disagree
editable pip install: pip install -e .
Bookmarking and if I'm ambitious will make a summary.
I find all of itertools to be worth a look.
for ...:
...
else:
....
Seriously, don't do that. I'll find you and I'll punch you.
I used this once in a leetcode problem and that’s it lol
The thing when we do alt+click and we can work on multiple cursor and edit all at a time..that was very surprising for me😅
That's not a feature of Python!
Totally useless, but stuff like this works in python.
_ = type("", (), {"ก็็็็็็็็็็็็็็็็็็็็": lambda x: print("hello %s" % x)})
_.ก็็็็็็็็็็็็็็็็็็็็("world")
Output: hello world
Do not show it Guido, he’ll have nervous breakdown if not heart attack :)
This reminds me…
Theorem of Unforeseen Reuse: For any function f written with a single intended use case, there exists at least one other programmer who will apply f in a way its author never anticipated.
ඞ
Big “switch” was when i fully understood and became comfortable with using standard dunder methods. Like you can define __getitem__ method in some class A and then do ‘for items in A:’