I have seen a "git for weights" implementation being demoed at a hackathon in 2024.
This was a runtime load balancing tool to balance deployment of finetunes across a fleet, based on load projections.
> , Snappy + Zstd produces a meaningful reduction in payload size compared with the LZ4 baseline.
The implementation was primarily derived from Google Stadia's data diff implementation (at least it got open sourced [2]) and other part was the columnar file format's floating point decompression (ALPv1[1]) ported to a GPU.
The ideas there are a local undo/redo log on SSDs, to switch the finetunes.
However, a lot of this is now out of date after the model serving layer has moved to a disaggregated distributed system instead of a single GPU with every layer loaded.
gopalv•4m ago
This was a runtime load balancing tool to balance deployment of finetunes across a fleet, based on load projections.
> , Snappy + Zstd produces a meaningful reduction in payload size compared with the LZ4 baseline.
The implementation was primarily derived from Google Stadia's data diff implementation (at least it got open sourced [2]) and other part was the columnar file format's floating point decompression (ALPv1[1]) ported to a GPU.
The ideas there are a local undo/redo log on SSDs, to switch the finetunes.
However, a lot of this is now out of date after the model serving layer has moved to a disaggregated distributed system instead of a single GPU with every layer loaded.
[1] - https://dl.acm.org/doi/10.1145/3626717
[2] - https://github.com/google/cdc-file-transfer/tree/main/cdc_rs...