I'm excited to share CompactLog, a Certificate Transparency log implementation written in Rust that explores an alternative approach to CT scalability challenges.
Certificate Transparency logs face immense scale - millions of certificates daily, with logs growing to hundreds of millions of entries and multiple terabytes of data. Traditional implementations often struggle with:
Database bottlenecks at multi-terabyte scale
Expensive hardware requirements for adequate read performance
Complex sharding and operational overhead
Read operations that can overwhelm the system
The CT community has been exploring different solutions to these challenges. One approach involves creating new APIs - moving from the traditional RFC 6962 dynamic endpoints to static, tile-based systems that can leverage CDNs and object storage more effectively. These new designs offer compelling benefits like efficient caching and horizontal scaling.
CompactLog explores a different question: Can we achieve similar scalability improvements while maintaining the existing RFC 6962 API?
Instead of changing the API, CompactLog rethinks the storage and implementation layers:
1. LSM-Tree Storage via SlateDB
Moving from relational databases to LSM-tree storage provides:
Natural fit for append-only workloads
Direct integration with object storage (S3, Azure Blob)
No artificial database size limitations
Cost-effective scaling with commodity storage
2. Zero Merge Delay Architecture
CompactLog eliminates merge delay entirely by incorporating certificates into the tree before issuing SCTs:
Submit → Add to tree → Issue SCT (all within 500ms)
This removes a common failure mode where logs fail to meet their merge delay commitments.
3. Smart Versioning Strategy
By versioning tree nodes only at Signed Tree Head boundaries rather than every update, storage overhead drops dramatically - potentially 1000x reduction for typical publishing intervals.
4. Performance
CompactLog achieves gigabits of API responses per second on commodity hardware, and is able to handle thousands of chain additions per second.
Eikon•8mo ago
Certificate Transparency logs face immense scale - millions of certificates daily, with logs growing to hundreds of millions of entries and multiple terabytes of data. Traditional implementations often struggle with:
The CT community has been exploring different solutions to these challenges. One approach involves creating new APIs - moving from the traditional RFC 6962 dynamic endpoints to static, tile-based systems that can leverage CDNs and object storage more effectively. These new designs offer compelling benefits like efficient caching and horizontal scaling.CompactLog explores a different question: Can we achieve similar scalability improvements while maintaining the existing RFC 6962 API?
Instead of changing the API, CompactLog rethinks the storage and implementation layers:
1. LSM-Tree Storage via SlateDB
Moving from relational databases to LSM-tree storage provides:
2. Zero Merge Delay ArchitectureCompactLog eliminates merge delay entirely by incorporating certificates into the tree before issuing SCTs:
This removes a common failure mode where logs fail to meet their merge delay commitments.3. Smart Versioning Strategy
By versioning tree nodes only at Signed Tree Head boundaries rather than every update, storage overhead drops dramatically - potentially 1000x reduction for typical publishing intervals.
4. Performance
CompactLog achieves gigabits of API responses per second on commodity hardware, and is able to handle thousands of chain additions per second.