However, I wonder whether the GPU's are a good fit for this to begin with.
Counterpoint: Xilinx side of the AMD shop has developed Alveo-series accelerators which used to be pretty basic SmartNIC platforms, but have since evolved to include A LOT more programmable logic and compute IP. You may have heard about these in video encoding applications, HFT, Blockchain stuff, what-have-you. A lot of it has to with AI stuff, see Versal[2]. Raw compute figures are often cited as "underwhelming," and it's unfortunate that so many pundits are mistaking the forest for the trees here. I don't think the AI tiles in these devices are really meant for end-to-end LLM inference, even though memory bandwidth in the high-end devices allows it.
The sauce is compute-in-network over fabrics.
Similarly to how PG-Strom would feed the GPU with relational data from disk, or network directly, many AI teams on the datacenter side are now experimenting with data movement, & intermediate computations (think K/V cache management) over 100/200/800+G fabrics. IMHO, compute-in-network is the MapReduce of this decade. Obviously, there's demand for it in the AI space, but a lot of it lends nicely to the more general-purpose applications, like databases. If you're into experimental networking like that, Corundum[3] by Alex Forencich is a great, perhaps the best, open source NIC design for up to 100G line rate. Some of the cards it supports also expose direct-attach NVMe's over MCIO for latency, and typically have as many as two, or four SFP28 ports for bandwidth.
This is a bit naive way to think about it, but it would have to do!
Postgres is not typically considered to "scale well," but oftentimes this is a statement about its tablespaces more than anything; it has foreign data[4] API, which is how you extend Postgres as single point-of-consumption, foregoing some transactional guarantees in the process. This is how pg_analytics[5] brings DuckDB to Postgres, or how Steampipe[6] similarly exposes many Cloud and SaaS applications. Depending on where you stand on this, the so-called alternative SQL engines may seem like moving in the wrong direction. Shrug.
[1] https://heterodb.github.io/pg-strom/
[2] https://xilinx.github.io/AVED/latest/AVED%2BOverview.html
[3] https://github.com/corundum/corundum
[4] https://wiki.postgresql.org/wiki/Foreign_data_wrappers
I think the GPU could be a great fit for OLAP, but when it comes to the nasty OLTP use cases the CPU will absolutely dominate.
Strictly serialized transaction processing facilities demand extremely low latency compute to achieve meaningful throughput. When the behavior of transaction B depends on transaction A being fully resolved, there are no magic tricks you can play anymore.
Consider that talking to L1 is at least 1,000x faster than talking to the GPU. Unless you can get a shitload of work done with each CPU-GPU message (and it is usually the case that you can), this penalty is horrifyingly crippling.
> Consider that talking to L1 is at least 1,000x faster than talking to the GPU.
This is largely true for "traditional" architectures, but s/GPU/TPU and s/L1/CMEM and suddenly this is no big deal anymore. I'd like Googlers to correct me here, but it seems well in line with classic MapReduce, and probably something that they're doing a lot outside of LLM inference... ads?
cpard•2h ago
It has been supported by engines like duckdb but the main serious use case of it I’m aware of is from Apache gluten where it is used to add Velox as the execution engine of Spark.
It’s an ambitious project and certainly has limitations but more projects like this are needed to push it forward.