there's some great stuff in here, particularly the direct-to-binary codegen, threads instead of processes and sync batching. you must have some intuitive notion about where the performance gains come from.

Would there be an advantage to reusing SIMD-optimized libraries for this?

Polars Rust is built on Arrow and packed_simd.

pola-rs/polars: https://github.com/pola-rs/polars

polars - Rust API docs: https://docs.pola.rs/api/rust/dev/polars/#simd

lancedb's data format; Lancedb/lance works with [Pandas, DuckDB, Polars, PyArrow,]; https://github.com/lancedb/lance

Narwhals' df interface (Python) https://narwhals-dev.github.io/narwhals/

substrait's portable query plans: https://substrait.io/ , https://github.com/ibis-project/ibis-substrait

Arrow RecordBatch, https://news.ycombinator.com/item?id=45495738#45546244

Port in OrioleDB next please. If our wish-granting really is at such a state. https://github.com/orioledb/orioledb

Port in OrioleDB next please. If our wish-granting really is at such a state. https://github.com/orioledb/orioledb

Great submission slipped by yesterday (!) on it's beta15 and beta16 released, on a big stability push. https://www.orioledb.com/blog/orioledb-beta15-16-stability https://news.ycombinator.com/item?id=49099108