What we need here is package caching in order to reduce download bandwidth and time, and its deduplication to avoid consuming storage space. This is important because worktrees are much more likely to share the same dependencies than unrelated projects. There may be language/toolchain specific solutions for this.
In case of nodejs/frontend projects, both npm and pnpm implement download caching. However, npm seems to copy the relevant modules to the project directory, duplicating the files in storage. Pnpm[1] on the other hand, uses a global cache and symlinks/hardlinks the relevant version of the relevant module to the project tree. This will save a lot of space when multiple worktrees are used.
In the python ecosystem, only conda[2] seems to provide this ability.
The sccache project [3] provides similar advantages to C, C++, Rust and CUDA projects. It caches the binary artifacts of compilation, so that can be used to speed up even first compilation of projects. By default, these caches can be shared between different projects on the same system. But with networked storage backends, sccache can be used to share build artifacts across a network and even enable parallel builds on multiple builds. (Added: Cargo seems to globally cache sources and deal with it on its own without using any links at all.)
Another important detail that might be worth consideration is the filesystem on which the worktrees reside. Copy-on-write filesystems like ZFS, btrfs and bcachefs provide automatic deduplication when copying files within a subvolume. I'm not sure what happens when worktrees are created by git. But they at least have tools to do manual deduplication (btrfs does). Perhaps we can just copy the package manager and build directories (like node_modules and build) from another worktree before running the development tools. Those tools are designed to handle stale cache anyway. So the wrong files wont be much of an issue for them. Perhaps tools like npm should look into making use of such filesystems, because you can get the same advantage as pnpm with practically no extra effort.
[1] https://pnpm.io/motivation#saving-disk-space
[2] https://docs.conda.io/projects/conda/en/latest/user-guide/co...
I was just repeating what chatgpt told me - that uv does the same thing as npm. The documentation wasn't clear about exactly what it does either. So I had to confirm it using strace. You are indeed correct. UV does hardlink the dependencies from the global cache to the project venv. So that's good!
There's just one caveat. Hardlinking is not possible if the project and the cache are on different file systems. Even btrfs subvolume mounts from a single physical volume is considered as different filesystems. I think pnpm quite happily symlinks it instead of hardlinking it when such a situation arises (I'm not sure though. Need to check that as well.) UV doesn't do that. It complains (prints a warning) and just copies the entire thing over to the venv. So you won't get that advantage with uv if the venv and the cache are on different filesystems (like in my system). I don't know how many developers actually deal with this problem.
conor_f•3mo ago