We built an open-source analytics database that allows companies to run real-time analytics workloads on top of open data formats (Iceberg, Parquet), instead of needing to store data in a closed-format datastore (e.g., ClickHouse, Druid…) for real-time analytics use cases.
We've built the engine from scratch in Rust and use Arrow for in-memory data processing, alongside a thread-per-core execution model to fully control the scheduling of CPU work. To achieve the speed we wanted, we had to build a custom memory allocator, tinker with the way JOIN and GROUP BY algorithms work to fully utilize the memory bandwidth of modern CPUs, and, in general, do a lot of optimization work to ensure we achieve superior performance compared to other real-time analytics engines, while remaining committed to using only Parquet / open data formats!
We're excited to share it with the community and hope others find it as useful as we have!
dkgs•3h ago
This is Gilad, one of the creators of Pivot
We built an open-source analytics database that allows companies to run real-time analytics workloads on top of open data formats (Iceberg, Parquet), instead of needing to store data in a closed-format datastore (e.g., ClickHouse, Druid…) for real-time analytics use cases.
We've built the engine from scratch in Rust and use Arrow for in-memory data processing, alongside a thread-per-core execution model to fully control the scheduling of CPU work. To achieve the speed we wanted, we had to build a custom memory allocator, tinker with the way JOIN and GROUP BY algorithms work to fully utilize the memory bandwidth of modern CPUs, and, in general, do a lot of optimization work to ensure we achieve superior performance compared to other real-time analytics engines, while remaining committed to using only Parquet / open data formats!
We're excited to share it with the community and hope others find it as useful as we have!