>black, optimized for losing heat inside

You're probably thinking this because of "black body radiation"[0], but room-temperature objects glow in the far IR, not in visible light. So what actually matters for radiant heat loss is to be 'black' in the far IR part of the spectrum, which is true for most non-metals.

Additionally, I doubt that radiative heat loss is a major cooling mode in clothing, vs convection and evaporation. I expect wearing black was more about protection from UV skin damage.

[0] https://en.wikipedia.org/wiki/Black_body_radiation