Hi all, I've been working for a while now on a fork of duperemove, which finds identical files/blocks and uses the filesystem's feature to deduplicate them.

My goal was to make it more convenient and faster, and more or less by accident it is now about 10 times faster in the deduplication phase.

The main trick is this: before sending data through the kernel's FIDEDUPERANGE ioctl, I simply read the data to prime the page cache. For some reason this results in an enormous speedup.

My fork has several other differences to the original, e.g the hashfile stores configuration, it's easier to automate, plenty of bug fixes (I backported the most severe ones), and a pretty UI.

I'd be interested in any feedback!