I was curious as to how that site works - it has a build script at https://github.com/hugovk/free-threaded-wheels/blob/cdae0b45... which checks the PyPI available file downloads for a package and looks for a bdist_wheel that matches this:
abi_tag = download["filename"].removesuffix(".whl").split("-")[-2]
if abi_tag.endswith("t") and abi_tag.startswith("cp31"):
has_free_threaded_wheel = TrueThis is the case with my own package which is on the hugovk list (apsw) which will cause the GIL to be re-enabled if you load it into a free threaded Python. The reason I provide a binary wheel is so that you don't have to keep separate GIL full and free threaded interpreters around. They have a different ABI so you can't use extensions compiled against one with the other.
Free threading is at the beginning of its journey. There is a *lot* of work to on all C code that works with Python objects, and the current documentation and tools are immature. It is especially the case that anyone doing Python concurrent object mutation can cause corruption and crashes if they try, and that more auditing and locking need to be done in the C code. Even modules in the standard library have only been partially updated.
You can see a lot details and discussion in the comments at https://news.ycombinator.com/item?id=45633311
> so that you don't have to keep separate GIL full and free threaded interpreters around
It means the user doesn't have to keep two Pythons around, install packages in both of them, etc.
It is also possible with the free threaded Python to keep the GIL disabled even if a package such as mine says it needs the GIL. And my package will indeed work just fine, until you supply it with mutable data and concurrently modify it in another thread.
Is it madness that other packages claim they support running without the GIL, yet it is possible to cause corruption and crashes just by writing concurrent Python code? That is the case with the standard library. Compiler thread sanitizers don't work with free threaded Python. Diligent code inspection by humans is the only way to update C code so far.
Free threading is at the beginning of the project. It works. You get slow downs in single threaded performance due to extra locking, and speedups in concurrent performance due to threading. But it is possible to cause corruption and crashes via Python code. Don't expose it to untrusted data and code.
But do investigate it and see what works well and what doesn't. See what code patterns are now possible. Help improve the tools and documentation. It had to start somewhere, and the current state is somewhere.
1) Use regular GIL Python and you get the highest levels of integrity and correctness of operation of my package
2) Use a free threaded Python, the GIL will be enabled at load time, and you get the highest levels of integrity and correctness
3) Use a free threaded Python, and set $PYTHON_GIL=0 or start with -Xgil=0 to keep the GIL disabled, and providing you do not do concurrent mutation of data provided to my package, you get the highest levels of integrity and correctness
BTW I did not randomly choose to provide the free threaded builds. I specifically asked the setuptools maintainers (under the Python Packaging Authority) how to prevent free threaded builds for PyPI. They encouraged me to do the free threaded builds so that a user doesn't to have maintain parallel regular and free threaded Python installations. And it allows option 3 above.
Wouldn't it be much better to just not support it if it's not supported?
1) Saying your package supports free threading, but it isn't safe - ie concurrent mutation can result in corruption and crashes
2) Allowing the package to be loaded into a free threaded Python, which immediately enables the GIL. Concurrent mutation does not result in corruption and crashes because of the GIL. The user doesn't have to maintain two Python installations. They can set the environment variable PYTHON_GIL=0 or start Python with -Xgil=0 which will keep the GIL disabled, and they will be fine if they avoid concurrent mutation.
I chose 2. The stdlib json package (along with many others) picked 1. Heck I'll guarantee that most that picked 1 aren't 100% safe either, because doing the changes is hard work, *every* case has to be covered, and tools like thread sanitizers don't work.
The reason I chose 2 is because I care about data integrity. I will eventually reach 1, but only once I can be certain the code is correct.
In practice, many built distributions contain binary artifacts (e.g. builds of CPython extensions). This differentiates them from source distributions, where you'd build the extension from source on your local machine.
This seems like a good opportunity to mention that that the wheel isn't required to contain any non-Python content, and indeed it's strongly preferred to provide one even (I would say "especially") when there is no meaningful "build step" (https://pradyunsg.me/blog/2022/12/31/wheels-are-faster-pure-...). There are of course abandoned legacy projects around from before the option was available, that may never get fixed. But developers of new code in pure Python should ensure they understand the system.
(I would strongly prefer if we could have a single type of package artifact with an optional post-install hook for building any needed non-Python content. Alas....)
This contrasts with "sdist", short for "source distribution", which potentially contains code in other languages which must be compiled on the end user's machine (which possibly involves downloading a corresponding build system and setting up a temporary build environment at install time).
Wheel and sdist distributions are successors to a legacy "egg" format. The PEP doesn't explain, but it's commonly held that the name "wheel" is a reference to wheels of cheese, based on PyPI's prior existence as "the cheese shop" (itself a reference to https://www.youtube.com/watch?v=Hz1JWzyvv8A). There might be some ancient discussion on the (now archived and inactive) mailing list to support this.
zahlman•3mo ago
psutil is a great project and I do have some future plans involving it.