Controlling a VOIP system, in which callers are challenged in different ways, is one approach. One potential challenge would be to issue random, non-sequential, and sparsely populated "extension" numbers, of say 10--20 digits (10 billion -- 100 quintillion values).

You'd offer a distinct value to each individual contact. Those contacts would be permitted to contact only that "extension". This creates a 1:1 pairing of caller to permitted contact number.

Coordinating generation and acceptance is probably the most challenging piece of this. Callers who might attempt contacts from multiple number is another.

(Which itself suggests assigning an extention to a set or range of numbers, e.g., a healthcare network, school or university system, or large business / organisation with which you engage frequently.)

Friends-and-family would have an easier challenge (whitelisted numbers, some in- or out-of-band validation, or simply being requested to enter the last four digits of their own number (robocall spoofs would probably fail that for at least some time.

Effectively, your single public "published" number serves as a gateway to your own telephony network, into which you can establish both valid numbers, restrictions on same, and challenges, as you see fit.