For the LAION classifier specifically, it's trained on-top of CLIP. The bottleneck for accuracy isn't the linear/non-linear readout, it's CLIP's specific knowledge of the world.

CLIP was trained with a contrastive learning objective, where all it has to learn is that images of the same category are closer to each other than images of other categories (e.g. two items are both dogs so they should be closer semantically, while say a dog and a computer are different and should be farther semantically.

As a result CLIP is really good at object definitions, but struggles at pretty much everything else. Modern multimodal LLM's perceptual modules are much stronger than CLIP; that's why we chose fine-tuning a LLM rather then training a MLP on-top of CLIP (which would be a direct iteration on the LAION aesthetic classifier).