National Security is a PITA, full of cutthroat sociopaths who would eat the SV VC-types for breakfast. That is a compliment, because the work they do is broadly dark and grimly necessary, at least at the levels of global geopolitics a lot of them are expected to operate at. I washed out in contracting for much the same reason this person kept "failing" polygraphs: honesty to the point of external perceptions of naivety. The types who excel in these sectors see folks like us as doormats or tissues, and react poorly when we catch them in the act and demand anything resembling respect because they know we're a threat to the entire establishment if we're allowed to succeed.

The point of polygraphs has always been about control, and folks who resist that sort of control are incidentally highlighting themselves as being uncontrollable to power alone. The books the author links are excellent starting points for understanding the true function of a polygraph, and why more places are outlawing them as a means of trying to diversify a deeply broken and hostile security apparatus by preventing it from being a "blind fools and sociopaths-only" club.

I knew a guy who did security clearance checking for the Three Letter Agencies for many years. He told be that if I ever had to do these interviews, I just need to pick good sounding lies and stick to them. He said it's the ones who try to be honest and introspective who get failed out.

Of course they do. And if you read the article in the OP you also realize why.

Polygraphs are an interrogation tactic, you can force a subject into a somewhat ridiculous procedure and ask them threatening questions, creating an disorientating situation. Afterwards you can accuse them of having "proven" that they are a liar. Polygraphs work, it just does not matter whether the machine is on or off.