Is it really fair to call these such algorithms "security through obscurity"?

By that logic, wouldn't real symmetric crypto also be considered the same, since it's still just math that similarly swaps the data with other bytes, based on an "obscured" password or key?