Templar AI has developed SparseLoCo, a distributed training algorithm that achieves extreme compression ratios (1-3% sparsity + 2-bit quantization) while outperforming existing methods like DiLoCo and DeMo on both loss and communication efficiency.
The Core Problem
Training LLMs across data centers or over the internet is bottlenecked by communication: as model scale grows, each synchronization can require transferring hundreds of gigabytes of pseudo-gradients. DiLoCo reduces the frequency of synchronizations, but the communication remains dense and large. This makes distributed training impractical for many scenarios, especially internet-scale collaboration.
Technical Approach
Our key insight: The infrequent communication of DiLoCo can be aggressively compressed via TOP-k sparsification while improving performance.
Algorithm highlights:
* Replace global momentum with per-replica error feedback
* Apply TOP-k magnitude compression (1-3% density) + 2-bit quantization to pseudo-gradients
* Maintain infrequent communication (H=15-250 steps) like DiLoCo
* Use chunked TOP-k for better parallelism and reduced index overhead
Results
Communication reduction: With >97× compression, SparseLoCo outperforms DiLoCo across all benchmarks. Sparse aggregation appears to provide regularization benefits beyond just compression.
Communication infrequency: Consistently outperforms DiLoCo across communication frequency ∈ {15, 30, 50, 100, 250} on 512M parameter models.
Real deployment: Currently running on Bittensor with a 70B model and 20 participants in the gather operation (out of many more total participants): 70 seconds communication with <500Mbps bandwidth. Our previous successful deployment of a medium sized (200B token) run of an 8B parameter model and 20 gather participants achieved communication average of 12 seconds vs 4.5 minutes compute time.
Key Technical Contributions
1. Local momentum approximation: Show that DiLoCo's global outer momentum can be well-approximated by local accumulators (>90% cosine similarity)
2. Error feedback as momentum: Demonstrate that TOP-k + error feedback naturally provides similar benefits to outer momentum
3. Sparse aggregation benefits: Find that sparse aggregation actually improves performance over dense methods—likely due to emphasis on high-saliency components
4. Extreme quantization: Error feedback enables 2-bit quantization without additional accumulators or performance drops
Implementation Details
* Chunked TOP-k (4096 elements/chunk) reduces index transmission overhead
* Custom index compression: 8.9, 6.6, 5.6 bits per value for different sparsity levels
* Drop-in replacement for DiLoCo all-reduce operations
* Compatible with existing distributed training frameworks
Limitations & Future Work
* Tested on 512M parameter models (though deployed on 8-70B)
* Chunk size optimization could be further explored
* Random-k performs significantly worse than TOP-k
This work makes distributed training viable over commodity internet connections and opens possibilities for global AI training collaborations that were previously bandwidth-prohibited.
synapz_org•5mo ago
Templar AI has developed SparseLoCo, a distributed training algorithm that achieves extreme compression ratios (1-3% sparsity + 2-bit quantization) while outperforming existing methods like DiLoCo and DeMo on both loss and communication efficiency.
The Core Problem
Training LLMs across data centers or over the internet is bottlenecked by communication: as model scale grows, each synchronization can require transferring hundreds of gigabytes of pseudo-gradients. DiLoCo reduces the frequency of synchronizations, but the communication remains dense and large. This makes distributed training impractical for many scenarios, especially internet-scale collaboration.
Technical Approach
Our key insight: The infrequent communication of DiLoCo can be aggressively compressed via TOP-k sparsification while improving performance.
Algorithm highlights:
* Replace global momentum with per-replica error feedback * Apply TOP-k magnitude compression (1-3% density) + 2-bit quantization to pseudo-gradients * Maintain infrequent communication (H=15-250 steps) like DiLoCo * Use chunked TOP-k for better parallelism and reduced index overhead
Results
Communication reduction: With >97× compression, SparseLoCo outperforms DiLoCo across all benchmarks. Sparse aggregation appears to provide regularization benefits beyond just compression.
Communication infrequency: Consistently outperforms DiLoCo across communication frequency ∈ {15, 30, 50, 100, 250} on 512M parameter models.
Real deployment: Currently running on Bittensor with a 70B model and 20 participants in the gather operation (out of many more total participants): 70 seconds communication with <500Mbps bandwidth. Our previous successful deployment of a medium sized (200B token) run of an 8B parameter model and 20 gather participants achieved communication average of 12 seconds vs 4.5 minutes compute time.
Key Technical Contributions
1. Local momentum approximation: Show that DiLoCo's global outer momentum can be well-approximated by local accumulators (>90% cosine similarity)
2. Error feedback as momentum: Demonstrate that TOP-k + error feedback naturally provides similar benefits to outer momentum
3. Sparse aggregation benefits: Find that sparse aggregation actually improves performance over dense methods—likely due to emphasis on high-saliency components
4. Extreme quantization: Error feedback enables 2-bit quantization without additional accumulators or performance drops
Implementation Details
* Chunked TOP-k (4096 elements/chunk) reduces index transmission overhead * Custom index compression: 8.9, 6.6, 5.6 bits per value for different sparsity levels * Drop-in replacement for DiLoCo all-reduce operations * Compatible with existing distributed training frameworks
Limitations & Future Work
* Tested on 512M parameter models (though deployed on 8-70B) * Chunk size optimization could be further explored * Random-k performs significantly worse than TOP-k
This work makes distributed training viable over commodity internet connections and opens possibilities for global AI training collaborations that were previously bandwidth-prohibited.