this is a nice and concise writeup. what's striking to me is that these techniques really have not changed in /years/. sure, precision has become slightly lower, spec decoding acceptance has gotten slightly better and the complexity of parallelism is trickier with mixture of experts. but no new concepts in a very long time!
the absolute most impactful improvements for inference comes at architecture design time. I firmly believe everyone who cares about impacting model efficiency should look there
philipkiely•29m ago
I think the biggest net new recent technique is P/D disaggregation. And that spec dec is very different now especially post DSpark/DFlash.
But overall yes the fundamentals of LLM performance optimization have been remarkably stable over the last few years.
datadrivenangel•7m ago
The author does not deeply mention that quality/intelligence is a third dimension here in addition to throughput and latency, and the frontier is jagged so quality and intelligence require bespoke benchmarks to evaluate tradeoffs for speed and cost.
brrrrrm•42m ago
the absolute most impactful improvements for inference comes at architecture design time. I firmly believe everyone who cares about impacting model efficiency should look there
philipkiely•29m ago
But overall yes the fundamentals of LLM performance optimization have been remarkably stable over the last few years.