Airport for DuckDB

https://airport.query.farm/

141•jonbaer•8mo ago

Comments

blef•8mo ago

This is a cool thought exercise to think that everything that we do in the data world can be done in SQL, from SQL. In a sense this is the MCPs but for the DuckDB world.

rustyconover•8mo ago

Thanks for taking the time to understand the philosophy of the extension.

simlevesque•8mo ago

last monday: https://news.ycombinator.com/item?id=44036343

mrbungie•8mo ago

I was almost going to build a lakehouse* with DuckDB because I low-key love it, easiest and strongest analytical engine I've found yet: scale from laptops to big metal, while being mostly out-of-core when doing sane stuff, and avoiding distributed computing for SQL in the process (looking at you Spark).

That is until I found out it does not support Iceberg writes[1], big nono as I would need another engine for inserts, and I want a simple stack :(. What a bummer.

[1] https://github.com/duckdb/duckdb_iceberg/issues/37

*that is what they are called now aren't they? I just can't follow the terms anymore haha.

mritchie712•8mo ago

it's coming. they already have hive style parquet writes. Iceberg is more complicated than that, but it's certainly doable.

mrbungie•8mo ago

Yeah, it just would be great if it already did so and I hope it supports Iceberg soon, as it would enable me to change expensive (and bad) engines like AWS Athena for something more manageable.

Don't get me wrong, I'm just being a tongue-in-check egotistical bastard data engineer from hell. DuckDB is a fine piece of software as it is, and those mantainers deserve heaven.

jeadie•8mo ago

This is one of the ideas behind using DuckDB in github.com/spiceai/spiceai

mrbungie•8mo ago

Looks very cool! I will take a look, tysm!

anentropic•8mo ago

That looks like an amazing "swiss army knife"...!

nicornk•8mo ago

Fivetran tried to upstream write support but it was not accepted https://github.com/duckdb/duckdb-iceberg/pull/95

shakna•8mo ago

That sounds less "not accepted" and more "will implement, rewrite required". It was only a couple months ago.

sukhavati•8mo ago

same here man, ended up going with trino explicitly for writing and data management and using chdb/duckdb to process data for front-ends etc (mostly ethereum data so chdb "support" for ui256 is quite important)

buremba•8mo ago

Not just for building a new one, it can also complement existing data-warehouse/lakehouses: https://github.com/buremba/universql

The flight extension is excellent as it removes the need to write C++ extensions and lets you use your favorite language to develop native DuckDB catalogs. It's straightforward to build data lake connectors and plug them in as a flight catalog, thanks to Airport!

benrutter•8mo ago

I'm curious, did you consider delta tables? Pretty sure duckdb supports them nicely. If you did, how come you chose not to go with them?

mrbungie•8mo ago

Afair (I might be wrong) AWS and a big chunk of the industry is promoting Iceberg over Delta. Delta is mostly backed up by Databricks.

the_optimist•8mo ago

What’s the situation where this is useful? Seems like ‘replace your remote duckDB instance—used to replace a DB server—with duckDB instance + a flight server (or a bunch of them!)’. Who has a problem for which this is the solution?

simlevesque•8mo ago

A Flight server paired with duckdb is a good way to get concurrent writes.

percevalve•8mo ago

if I got it correct, it helps connect to a wide array of backends, and even function calling.

You can then directly use SQL to work with data from all those at the same time.

The working assumptions then becomes that SQL is a dialect that has a wider adoption then python for example...

Making an educated guess here.

r3tr0•8mo ago

I love duck db. We use it a ton for indexing and organizing system / kernel level metrics exported by eBPF.

Check out our sandbox:

https://yeet.cx/play

rubenvanwyk•8mo ago

Does this mean the data source and destination both have to set up flight servers? I imagine then this won’t be useful for integration of third-party services.

rustyconover•8mo ago

Only the data source.

k_bx•8mo ago

Not clear. Will this allow loading ipc files in DuckDB finally? That's been my biggest issue, since I use IPC files for append operations before I turn them into parquet files.

rustyconover•8mo ago

That’s possible with the arrow extension today.

k_bx•8mo ago

I was sure it supported .arrow but not the streaming .ipc format, but will re-check when I have a chance

samansmink•8mo ago

It was not supported for quite a while indeed, but now there's https://duckdb.org/2025/05/23/arrow-ipc-support-in-duckdb.ht...

vkaku•8mo ago

This is very nice. I also love the fuzzycomplete and lindel from the same org/authors.

code_biologist•8mo ago

fuzzycomplete - https://github.com/Query-farm/fuzzycomplete "This fuzzycomplete extension serves as an alternative to DuckDB's autocomplete extension, with several key differences: ..."

lindel - https://github.com/Query-farm/lindel "This lindel extension adds functions for the linearization and delinearization of numeric arrays in DuckDB. It allows you to order multi-dimensional data using space-filling curves. ... Linearization maps multi-dimensional data into a one-dimensional sequence while preserving locality, enhancing the efficiency of data structures and algorithms for spatial data, such as in databases, GIS, and memory caches."

rustyconover•8mo ago

Thanks for the compliments!

We Mourn Our Craft

Speed up responses with fast mode

U.S. Jobs Disappear at Fastest January Pace Since Great Recession

Hoot: Scheme on WebAssembly

Stories from 25 Years of Software Development

OpenCiv3: Open-source, cross-platform reimagining of Civilization III

Al Lowe on model trains, funny deaths and working with Disney

Brookhaven Lab's RHIC Concludes 25-Year Run with Final Collisions

The AI boom is causing shortages everywhere else

The Waymo World Model

Reinforcement Learning from Human Feedback

I Write Games in C (yes, C)

Start all of your commands with a comma (2009)

SectorC: A C Compiler in 512 bytes

Vocal Guide – belt sing without killing yourself

France's homegrown open source online office suite

Coding agents have replaced every framework I used

A Fresh Look at IBM 3270 Information Display System

Selection Rather Than Prediction

History and Timeline of the Proco Rat Pedal (2021)

72M Points of Interest

Unseen Footage of Atari Battlezone Arcade Cabinet Production

Show HN: I saw this cool navigation reveal, so I made a simple HTML+CSS version

Where did all the starships go?

Show HN: Look Ma, No Linux: Shell, App Installer, Vi, Cc on ESP32-S3 / BreezyBox

Show HN: Kappal – CLI to Run Docker Compose YML on Kubernetes for Local Dev

Learning from context is harder than we thought

Monty: A minimal, secure Python interpreter written in Rust for use by AI

Making geo joins faster with H3 indexes

Software factories and the agentic moment

We Mourn Our Craft

Speed up responses with fast mode

U.S. Jobs Disappear at Fastest January Pace Since Great Recession

Hoot: Scheme on WebAssembly

Stories from 25 Years of Software Development

OpenCiv3: Open-source, cross-platform reimagining of Civilization III

Al Lowe on model trains, funny deaths and working with Disney

Brookhaven Lab's RHIC Concludes 25-Year Run with Final Collisions

The AI boom is causing shortages everywhere else

The Waymo World Model

Reinforcement Learning from Human Feedback

I Write Games in C (yes, C)

Start all of your commands with a comma (2009)

SectorC: A C Compiler in 512 bytes

Vocal Guide – belt sing without killing yourself

France's homegrown open source online office suite

Coding agents have replaced every framework I used

A Fresh Look at IBM 3270 Information Display System

Selection Rather Than Prediction

History and Timeline of the Proco Rat Pedal (2021)

72M Points of Interest

Unseen Footage of Atari Battlezone Arcade Cabinet Production

Show HN: I saw this cool navigation reveal, so I made a simple HTML+CSS version

Where did all the starships go?

Show HN: Look Ma, No Linux: Shell, App Installer, Vi, Cc on ESP32-S3 / BreezyBox

Show HN: Kappal – CLI to Run Docker Compose YML on Kubernetes for Local Dev

Learning from context is harder than we thought

Monty: A minimal, secure Python interpreter written in Rust for use by AI

Making geo joins faster with H3 indexes

Software factories and the agentic moment

Airport for DuckDB

Comments