Chew here, CTO at [Alphaus](https://alphaus.cloud/en/). We've been running a similar version of Luna at Alphaus, to cache really huge CSVs and PARQUETs in memory (hundreds of GBs per account across multiple files). It's written in Go, and uses DuckDB and Apache Arrow behind the scenes.
We're now at a point where a single VM's memory is really not enough to hold these huge files (the really hi-mem VMs are not yet available in our region) so we're thinking of distributing the cache across multiple hi-mem VMs (with disk spill-over). I've been "itching" to rewrite this software using a non-GC language, and so I thought it's a good timing to do an OSS version of it. I did consider C++ (and Zig) but I think Rust is a good fit here.
I've looked at [Ballista](https://github.com/apache/datafusion-ballista) actually as I'm quite familiar with DataFusion, and of course, MotherDuck, but the amount of custom duct-tapes we do behind the scenes made me think it's going to be worth the effort.
And so, sharing to you, "Luna". Still alpha-level, and still a lot of things missing, but the main goal is an easy-to-use, easy-to-deploy, columnar in-memory SQL server process, leveraging DuckDB and Apache Arrow, with more focus on cloud and object storage data. And a distributed/cluster version of Luna as well; sort of alternative to Ballista.