frontpage.
newsnewestaskshowjobs

Made with ♥ by @iamnishanth

Open Source @Github

fp.

Open in hackernews

Loggr: Processing 250M logs in 11.5s on a laptop with on-the-fly 5× compression

https://medium.com/@fgauthier_36718/loggr-processing-250m-logs-in-11-5s-on-a-laptop-with-on-the-fly-5-compression-7903b3f941d4
1•fsg-swl•3mo ago

Comments

fsg-swl•3mo ago
We built Loggr, a 170KB C logging library that preprocesses and compresses logs at line rate.

Key results on a Lenovo P14s (Ryzen 5 Pro, NVMe): - 250M logs in 11.52 seconds (21.71M logs/sec) to disk - ~5× end-to-end compression (preprocessing + LZ4) - 105MB RAM footprint, 0 logs lost

The article details: - Architecture with smart preprocessing + temporal caching - Benchmark methodology with standard/adversarial workloads - Core API documentation and configuration parameters - Tradeoffs analysis and limitations - Complete telemetry data and reproduction guidelines

We're looking for technical feedback and use case suggestions.