He sent a $1,000 check as "collateral" to some online "bank" to get a $10,000 "loan". I explained to him that this isn't how any of this works and he just paused and said, "Huh, so should put a stop payment on the second check?" He had sent them another $1,000 as some sort of second round of collateral for the loan or some other nonsense.

And I've known dumber people than him.

I assume that the "people who fall for this stuff" are "people like me". Even the best among us are only statistically less likely to get snookered, not immune.

I think you nailed it. I made a comment about the new "religion" of games in the current youth in another discussion a while back. It really seems to me that there is an incredibly strong illogical loyalty to game companies in the current youth. So much that they ignore bad actors assuming that they must be just another game company, even after they get scammed. I'm guessing it has something to do with zero real value nature of loot box and skin gambling that is prevalent.

And that threshold will move over time, and as you say, depending on conditions. Even our personal defenses should not be predicated on being perfect all the time, and that's completely impractical at a societal level. And I worry about AI making it practical to target more people directly at scale and raise the sophistication on their end over the next 10-20 years.

Today I think I can say with a straight face that I'm quite sophisticated against this sort of scam. But the day will come, unless something gets me earlier, when my children will have to take my email account away if they want to inherit anything. Old age makes fools of us all. And who knows if something will get me earlier some other way. Unfortunately, invincible confidence in the here and now won't protect me.