Improvement in production processes and materials (e.g. magnetic coatings) allowing smaller tracks and smaller more precisely positioned heads. The 3 1/2 floppy dates back to 1983, the high-density 1.44MB to 1986, the Zip drive was released in 1994.

A “super high density” 20 MB floppy had already been attempted in 1990, and the LS-120, which had the exact same dimensions as a 3.5” floppy (and could read those), launched in 1996, so it was not really exceptional at 6 doubling in 8 years from the 1.44MB floppy.

Also it was expensive, part of that was the lower scale and lack of competition but the increased production requirements were also a factor, Zip drives and media had tighter tolerances.

The click of death was because when the head got misaligned the drive would return it to the home position, if part of the drive had failed the head would never realign so the drive would keep trying, producing a characteristic clicking sound. HDDs can develop the same, but it’s less common than it was on Zip drives. The tighter tolerance were most likely a factor, it was more likely for a zip to age out of tolerance and develop terminal misalignment.

A related technology with a name that already answers your question.

But only a few years later, as you say, USB thumb drives were making Zip drives irrelevant.

I think I only saw two computers with ZIP drives in the wild, so to say. Which created its own anti-network effects: if there is no expectation of the other party having a compatible drive, you will either have to do with floppies, or maybe carry an external HDD with you, but you won't use an exotic and expensive format. Pretty much the only plausible use was backup.

I still believe that with a more aggressive price policy, ZIPs could have conquered a lot of territory in the 1990s. But 200/20 USD was just too much money to spend. Not just in the post-Soviet bloc, but almost everywhere in Asia, too. Big markets lost because of the cost, and therefore a chance to entrench the standard worldwide.

I'm not surprised, considering they were expensive. That doesn't mean they weren't popular in the west.