That got me thinking about something in between. So I built ForesightJS in the open. Instead of prefetching onHover, it prefetches onIntent. If the mouse is clearly moving toward a link, it gets prefetched before the hover even happens. Same with keyboard navigation. If a user is a few tab stops away from a target, ForesightJS will kick in and prefetch it early.
I completely agree that this might not be for everyone, however if you do want to squeeze out those extra ms you might want to give it a try!