The way they quantify this is: they pick a biggish set of knots that are known all to be distinct from one another. They then compute their invariant for each of those knots. A knot invariant successfully distinguishes them all from one another precisely when it takes different values for all of the knots. So they count the number of different values their invariant takes, and subtract it from the number of knots. They call this the "separation deficit": the smaller the better.

They compare their invariant with some already-known ones, taking "all knots that can be drawn in the plane with <= 15 crossing points" as their set of knots. There are about 300,000 of these.

One of the best-known knot invariants is the so-called Alexander polynomial. That's in row 3 of Table 5.1, and its "separation deficit" for those knots is on the order of 200k. That is, these 300k knots have between them only about 100k different Alexander polynomials; if you pick a random smallish knot and compute its Alexander polynomial then handwavily you should expect that there are two other different smallish knots with the same Alexander polynomial.

Another knot polynomial, which does a better job of distinguishing different knots, is the so-called HOMFLY polynomial. (Why the weird name? It comes from the initials of the six authors of the paper announcing its discovery.) That's row 7, showing a deficit of about 75k. That suggests, even more handwavily, that if you pick a random smallish knot and compute its HOMFLY polynomial, there's about a 1/3 chance that there's another smallish knot with the same HOMFLY polynomial. Still not great.

A rather different sort of invariant is the hyperbolic volume of the complement of the knot. That is: if you take all of space minus the knot then there's a certain nice way to define distances and volumes and things in the left-over space; the whole of space-minus-the-knot turns out then to have a finite volume, and perhaps surprisingly deforming the knot doesn't change that volume. So that's another knot invariant, and it turns out to be better at distinguishing knots from one another than the polynomials mentioned above, on the order as 2x better than the HOMFLY polynomial.

This paper's invariant (which is a pair of polynomials) does about 6x better than what you get by looking at the Alexander polynomial, the HOMFLY polynomial, the hyperbolic volume, and a few other invariants I didn't mention above, all together. Its "separation deficit" on this set of ~300k knots is about 7000. If you pick a random smallish knot, there's only about a 2% chance that some other knot has the same value of this paper's invariant.

(Reminder that all this business about probabilities is super-handwavy. Actually, that probability might be anywhere from about 2% to about 4% depending on exactly how the values of the invariant are distributed.)

Now, all of this is purely empirical and looks only at smallish knots. So far as I know they haven't proved any theorems like "our invariants do a better job than the hyperbolic volume for knots with <= N crossings, for all N". I think such theorems are very hard to come by.

They don't, to be clear, claim that their invariant is the best at distinguishing different knots from one another. For instance, they mention another set of knot polynomials that does a better job but is (so they say) much more troublesome to compute for a given knot.

Richard Crowell and Ralph Fox, "An Introduction to Knot Theory" (1963) - an old book which doesn't recent developments. The Dover edition is only $15.95: https://store.doverpublications.com/products/9780486468945?_...

Ralph Fox, "A Quick Trip Through Knot Theory" (1962). The original paper was a chapter in M. K. Fort's "Topology of 3-Manifolds" collection, but I think you can find copies online.

Dale Rolfsen, "Knots and Links". A newer book and one of the best if you know some (algebraic) topology. He argue for considering the Alexander invariant (the homology of the infinite cyclic cover) rather than the Alexander polynomial. I don't have the newest edition, so I don't know how far it goes in terms of recent developments.

The book by Burde and Zeischang has this stuff and more, but it's more advanced.

I'm not a knot theorist, so I don't know about newer books - maybe someone else has better recommendations.