The memory capacity wall has been coming at us at high speed in plain sight for decades. We've been ignoring it.

But still it is about GiB instead of KiB.

This is driven by AI and at a certain point you just need a minimum amount of memory to load all the floating points representing Neural Net weights.

Not everyone is using baidu?

I have this wild theory that this has not just todo with AI but with weapons production overall.

Why use less memory when you can just have more LLM?

If training and inference is hardware constrained, and you can train and server better and bigger models on the same hardware with memory optimizations, that's exactly what I would expect companies to do.

Demand outstrips supply. Efficient software is great, but it doesn’t directly resolve insufficient supply of hardware.

How?