What's the point of the relu in the loss function? Its inputs are nonnegative anyway.

You can also imagine a similar thing on binary vectors. There two vectors are "orthogonal" if they share no bits that are one. So if you can encode huge number of concepts using only small number of bits in modestly sized vectors, and most of them will be orthogonal.