That is until I found out it does not support Iceberg writes[1], big nono as I would need another engine for inserts, and I want a simple stack :(. What a bummer.
[1] https://github.com/duckdb/duckdb_iceberg/issues/37
*that is what they are called now aren't they? I just can't follow the terms anymore haha.
Don't get me wrong, I'm just being a tongue-in-check egotistical bastard data engineer from hell. DuckDB is a fine piece of software as it is, and those mantainers deserve heaven.
The flight extension is excellent as it removes the need to write C++ extensions and lets you use your favorite language to develop native DuckDB catalogs. It's straightforward to build data lake connectors and plug them in as a flight catalog, thanks to Airport!
You can then directly use SQL to work with data from all those at the same time.
The working assumptions then becomes that SQL is a dialect that has a wider adoption then python for example...
Making an educated guess here.
Check out our sandbox:
lindel - https://github.com/Query-farm/lindel "This lindel extension adds functions for the linearization and delinearization of numeric arrays in DuckDB. It allows you to order multi-dimensional data using space-filling curves. ... Linearization maps multi-dimensional data into a one-dimensional sequence while preserving locality, enhancing the efficiency of data structures and algorithms for spatial data, such as in databases, GIS, and memory caches."
blef•8mo ago
rustyconover•8mo ago