(there is now a long queue id option which adds a time component)
The combination of st_dev and st_ino from stat() should be unique on a machine, while the device remains mounted and the file continues to exist.
If the file is deleted, a different file might get the inode, and if a device is unmounted, another device might get the device id.
It should, but it seems no longer to be the case. I believe there was an attempt to get a sysctl flag in to force the kernel to return the same inode for all files to see what breaks.
I don't think all filesystems guarantee this. Especially network filesystems.
Even if inodes still work nowadays they will eventually run into issues a few years down the line.
find:
* https://cgit.git.savannah.gnu.org/cgit/findutils.git/tree/fi...
* https://cgit.git.savannah.gnu.org/cgit/findutils.git/tree/fi...
tar:
* https://cgit.git.savannah.gnu.org/cgit/tar.git/tree/src/crea...
* https://cgit.git.savannah.gnu.org/cgit/tar.git/tree/src/name...
* https://cgit.git.savannah.gnu.org/cgit/tar.git/tree/src/incr...
In particular, I'm intrigued by the comment in the last link:
/* With NFS, the same file can have two different devices
if an NFS directory is mounted in multiple locations,
which is relatively common when automounting.
To avoid spurious incremental redumping of
directories, consider all NFS devices as equal,
relying on the i-node to establish differences. */
> Well, the fact that it hits snapshots, shows that the real problem is just "tar does stupid things that it shouldn't do".
> Yes, inode numbers used to be special, and there's history behind it. But we should basically try very hard to walk away from that broken history.
> An inode number just isn't a unique descriptor any more. We're not living in the 1970s, and filesystems have changed.
You might still get away with it most of the time today, but it's causing more and more issues.
[1]: https://lkml.iu.edu/hypermail/linux/kernel/2401.3/04127.html
If it's not the 1970s anymore, then update the POSIX standard with a solution that works for all OSes (including the BSDs) and can be relied upon. Definitely don't suggest a Linux-only solution for a Linux-only problem.
> The st_ino and st_dev fields taken together uniquely identify the file within the system.
It says exactly what it ought to say.
As an example, it offers that the identity of a file that's deleted could be reused.
> A file identity is uniquely determined by the combination of st_dev and st_ino. At any given time in a system, distinct files shall have distinct file identities; hard links to the same file shall have the same file identity. Over time, these file identities can be reused for different files. For example, the st_ino value can be reused after the last link to a file is unlinked and the space occupied by the file has been freed, and the st_dev value associated with a file system can be reused if that file system is detached ("unmounted") and another is attached ("mounted").
I still think POSIX says exactly what it needs to say, and Linux ought to either comply with it, or lead the standardisation process on what should be done instead.
Don't say "tar is old". Tar's problems with Linux are the same problems that find, zip, rsync, cp and all other fs walking programs have. If memorising st_dev and st_ino are no good, tell us what cross-platform approach should be taken instead.
"A centimeter is no longer based on anything and has an unpredictable length. Rulers always did stupid things relying on that assumption."
Sure, within the bounds of what's documented you are right. However tar is going beyond what either standard or Linux guarantee so a lot of bets are off.
The guarantee that tar wants is not given by any FS that recycles inodes and most importantly, tar already completely disregards the file-system locality when network drives are involved.
The actual issue here is that both tar and Linux are just in a tough situation because a) the POSIX spec is problematic b) no alternative API exists today. Something has to give.
You are absolutely right that ideally there is an update to the POSIX standard. But things like this take time and it's also not necessarily clear yet what the right path here is going forward. You can consider a lot of what is currently taking place as an experiment to push the envelope.
As for if this is a Linux specific problem I'm not sure. I'm not sufficiently familiar with the situation on other operating systems to know what conversations are taking place there.
The core problem is that, because of the ability of filesystems to effectively contain other filesystems within them, the number of bits to uniquely identify a file within a filesystem is not a constant number across different filesystem types. It's a harder problem on Linux because Linux is also full of filesystems that aren't really filesystems, where trying to come up with a persistent, unique identifier for people to use is a lot more bother than it's really worth.
According to who? Clearly there are user space utilities that need a (somewhat) persistent identifier to work correctly.
Unfortunately, it doesn't even seem to be exposed in statx (!). There's change_cookie, but that's different.
If anyone wants to submit a patch for this, I'll be happy to review it.
It's a bit of both. inodes are conflating two things in a way. They are used by the file system to identify a record but they are _also_ exposed in APIs that are really cross file system (and it comes to a head in case of network file systems or overlayfs).
What's a more realistic path is to make inodes just an FS thing, let it do it's thing, and then create a set of APIs that is not relying on inodes as much. Linux for instance is trying to move towards file handles as being that API layer.
And really, it's not even super necessary. 64-bit inode numbers already exist in modern file systems. You don't need UUIDs to have unique IDs forever: you'll never run out of 64-bit integers. But the problem wasn't really ever that you'd run out, the problem is in the way they're handled.
You could do it like Java's Object.identityHashCode() and allocate durable IDs only on demand.
> If an inode number is 32 bits today, then UUIDs would take up four times the space.
We probably waste more space on filesystems that lack tail-packing.
> I'd also guess that the cost of hashing the UUIDs is significant enough that you'd see a user-visible performance hit.
We're hashing filenames for H-tree indexing anyway, aren't we?
> you'll never run out of 64-bit integers
Yeah, but with 128-bit ones you'll additionally never collide.
dd if=/dev/sda of=/dev/sdb would like a word with you.
The two real issues here are 1) inode numbers are no longer sequentially allocated as back in the 1970-80's, and 2) an inode number in a modern file system usually carries extra significant information.
In XFS, the inode number is derived from the disk address (block address) of the inode within the file system. Specifically, the inode number encodes:
• The allocation group (AG) number.
• The block offset within the AG.
• The inode’s index within its block.
Then, there is an issue of the finite size (or width) of the inode field (64 bits, 128 bits etc). On an exascale file system with a very high file turnover and a huge number of files, at some point a previously allocated inode will have to be recycled, irrespective of whether it was a pseudorandom or a calculated number. It is not a problem for most installations as the exascale is not there, but I don't think the problem can be solved using non-esoteric approaches.
> The root inode is the root of the file system. Inode 0 can't be used for normal purposes and historically bad blocks were linked to inode 1 (inode 1 is no longer used for this purpose; however, numerous dump tapes make this assumption, so we are stuck with it). Thus the root inode is 2.
This is also echoed on the wikipedia page for it.
The linux Kernel also has this comment for why it does not dish out that inode for shmem for instance:
> Userspace may rely on the the inode number being non-zero. For example, glibc simply ignores files with zero i_ino in unlink() and other places.
On macOS it's pretty clear that inode 0 is reserved:
> Users of getdirentries() should skip entries with d_fileno = 0, as such entries represent files which have been deleted but not yet removed from the directory entry
A file descriptor can't be -1 but it's not 100% clear whether POSIX bans other negative numbers. So Rust's stdlib only bans -1 (for a space optimization) while still allowing for e.g. -2.
It turns out that glibc readdir() assumes inode zero doesn't happen and the files are "invisible" to anything using libc. But you can call getdents() directly and see them.
I actually ran into this on a production machine once a few years ago, a service couldn't restart because a directory appeared to be "stuck" because it had one of these invisible zero inode files in it. It was very amusing, I figured it out by spotting the invisible filename in the strace output from getdents().
Am I missing something or is this just evil? I guess I'm taking it too seriously.
For example, knowing that inodes 0 are technically possible will tell you to not rely on inode 0 as a special magic number. It's when parts of the system disagree about details like this that you get issues, not when you learn about them.
Animats•1d ago