How we made Python's packaging library 3x faster

https://iscinumpy.dev/post/packaging-faster/

88•rbanffy•3w ago

Comments

zahlman•3w ago

Previously: https://news.ycombinator.com/item?id=46557542

djoldman•2w ago

> _canonicalize_table = str.maketrans( "ABCDEFGHIJKLMNOPQRSTUVWXYZ_.", "abcdefghijklmnopqrstuvwxyz--", )

> ...

> value = name.translate(_canonicalize_table)

> while "--" in value:

> value = value.replace("--", "-")

translate can be wildly fast compared to some commonly used regexes or replacements.

teaearlgraycold•2w ago

I would expect however that a regex replacement would be much faster than your N^2 while loop.

notpushkin•2w ago

It would be, if it was a common situation.

This loop handles cases like `eggtools._spam` → `eggtools-spam`, which is probably rare (I guess it’s for packages that export namespaced modules, and you probably don’t want to export _private modules; sorry in advance for non-pythonic terminology). Having more than two separator characters in a row is even more unusual.

dgrunwald•2w ago

That loop isn't N²: if there are long sequences of dashes, every iteration will cut the lengths of those sequences in half. So the loop has at most lg(N) iterations, for a O(N*lg(N)) total runtime.

est•2w ago

I am curious, why not .lower().translate('_.', '--')

fwip•2w ago

.lower() has to handle Unicode, right? I imagine the giant tables slow it down a bit.

mort96•2w ago

It's so annoying how so many languages lack a basic "ASCII lowercase" and "ASCII uppercase" function. All the Unicode logic is not only unnecessary, but actively unwanted, when you e.g want to change the case of a hex encoded string or do normalization on some machine generated ASCII-only output.

tracker1•2w ago

I'll say, C#'s .ToLowerInvariant, etc. are pretty nice when you need them.

est•2w ago

> It's so annoying how so many languages lack a basic "ASCII lowercase" and "ASCII uppercase" function

How about b''.lower() ?

mort96•2w ago

What if I have a string and not a byte string?

ltbarcly3•2w ago

Misleading title, they didn't make the packaging library 3x faster, they made reading one attribute of a package 3x faster. The whole library is still very, very slow compared to alternatives.

imtringued•2w ago

Unrelated, but I personally am not satisfied with the performance of Panda's XLSX export. As you can see here [0], the code does really strange things. It takes cell.style and throws it into json.dumps() to generate a key for a dictionary so that they can cache the XlsxStyler.convert(cell.style) result. Except, the vast majority of cells do not have any styling whatsoever, so json.dumps is producing the string "null", which is then used to lookup None. The low hanging fruit are jaw dropping. You can easily speed up the code 10%+ by adding a simple check "if cell.style is not None or fmt is not None:" and switching from json.dumps(cell.style) to str(cell.style). If I wanted an easy weekend project that positively impacts many people this is what I'd work on.

[0] https://github.com/pandas-dev/pandas/blob/main/pandas/io/exc...

rthz•2w ago

Have you tried opening an issue about it? Maybe someone would be happy to work on it. I concur that Excel parsing is rather slow.

Show HN: LLM of Babel

A modern iperf3 alternative with a live TUI, multi-client server, QUIC support

Famfamfam Silk icons – also with CSS spritesheet

Apple is the only Big Tech company whose capex declined last quarter

Reverse-Engineering Raiders of the Lost Ark for the Atari 2600

Show HN: Deterministic NDJSON audit logs – v1.2 update (structural gaps)

The Greater Copenhagen Region could be your friend's next career move

Do Not Confirm – Fiction by OpenClaw

The Analytical Profile of Peas

Hallucinations in GPT5 – Can models say "I don't know" (June 2025)

What AI is good for, according to developers

OpenAI might pivot to the "most addictive digital friend" or face extinction

Show HN: Know how your SaaS is doing in 30 seconds

ClawdBot Ordered Me Lunch

What the News media thinks about your Indian stock investments

Running Lua on a tiny console from 2001

Google and Microsoft Paying Creators $500K+ to Promote AI Tools

New filtration technology could be game-changer in removal of PFAS

Show HN: I saw this cool navigation reveal, so I made a simple HTML+CSS version

Kinda Surprised by Seadance2's Moderation

I Write Games in C (yes, C)

Django scales. Stop blaming the framework (part 1 of 3)

Malwarebytes Is Now in ChatGPT

Thoughts on the job market in the age of LLMs

Show HN: Stacky – certain block game clone

AIII: A public benchmark for AI narrative and political independence

SectorC: A C Compiler in 512 bytes

The API Is a Dead End; Machines Need a Labor Economy

Digital Iris [video]

New wave of GLP-1 drugs is coming–and they're stronger than Wegovy and Zepbound