This seems like a fundamental problem with juries. If everyone knows you, most people detest you, and you can only be tried by a jury which is indifferent to you, is that jury truly impartial? What sort of people could be in Hitler's jury, say, to take it to an extreme?

You want a jury with good judgment. Maybe people indifferent to the publicly known actions of the accused are that way precisely because they lack good judgment. Yes, they need to judge the case in question, not everything the accused has ever done, but perhaps indifference to those facts extraneous to the case indicates they will judge pertinent facts in a way the public at large wouldn't recognize as just.