APFS does suck though, don't get me wrong
edit: I just tested, 36s for an `npm i` on a decently large project I have locally, 46s with an overlay fs
I mean, I don't use NPM, but I recently copied over some small-ish raw Git repositories between machines, and had several WTF moments when I looked at what was actually going over the wire.
Instead of refactoring two of the planet's most widely-used 'consumer' file systems, perhaps https://www.sqlite.org/appfileformat.html should be seen as a more reasonable way forward?
This is because C: has additional file system filters for system integrity that are not present on other drives; and also because C: has 8.3 name compatibility enabled by default, but additional NTFS partitions have that disabled. (so if your filenames are longer than the 8.3 format, Windows has to write twice the number of directory entries)
But in the end both npm and git ends up having mass writing files in their use cases, regardless of meta data that could be put in a sqlite-like db. Making things faster safely really implies having those apps operating on some OS features that would allow of acquiring lock and committing semantics on fs subtrees or equivalent.
- pathological filesystem behavior => git/npm/whoever should use a SQLite DB as a flat file rather than having multiple files.
- little to no pushback
I would suspect that this is related to to the fact that reading/writing dirs have no atomic semantics in POSIX fs, and same directories are easily read/written simultaneously by unrelated processes, but surely is cause to edge cases bugs, especially in a world with backup system based on fs snapshots.
This is a sad state of affairs.
https://gregoryszorc.com/blog/2018/10/29/global-kernel-locks...
Couldn’t these tests be instrumented to give a deeper dive into what may be happening here?
systemz•1h ago
tuyiown•1h ago
udev•56m ago