The Fill function for interpolation is really nice, but there's so many different ways to perform interpolation I'd like a lot more detail on what methods are available and which is used by default.
tanelpoder•4mo ago
Indeed. Previously, when computing latency histograms from sampled events, I've abused DuckDB SQL cross join/cartesian products to generate zero values for "empty buckets" within a single SQL statement [1]. But makes the SQL unnecessarily complex & slower too, so I quickly moved that empty bucket rendering functionality to the frontend...
Feature author here. The new FILL uses linear interpolation (and extrapolation at the ends). There were a bunch of other algorithms requested (e.g., cubic splines), but the syntax was not clear.
I think the real solution is to add support for non-aggregate window functions in the catalogue. Right now, they are all hard-coded, but if they were extensible, there are a number applications in addition to filling that could be supported.
geysersam•4mo ago
You can always manually define your interpolation:
coalesce(x, (lag(x, 1)+lag(x, -1))/2)
is kind of the same as fill.
From there you can go crazy and define whatever insane interpolations you want.
One limitation is that you can't distinguish the data boundaries from missing values (nulls at the edges vs. nulls in the bulk).
I guess another limitation is if you have multiple nulls after each other...
e12e•4mo ago
> Starting with this release, every other DuckDB version is going to be a Long Term Support (LTS) edition. For LTS DuckDB versions, community support will last a year after the release (for now). DuckDB Labs is also starting to offer support for older LTS versions after their community support has expired.
A fixed date is great - but it's difficult to see a single year as "long term" support.
Will be interesting to what commercial support options will emerge.
datadrivenangel•4mo ago
LTS is a move in the right direction. One year is somewhat short, but with no backwards breaking changes that's fine.
zui•4mo ago
Awesome that the iceberg write is now possible from duckdb!
ZeroCool2u•4mo ago
tanelpoder•4mo ago
[1]: Example DuckDB SQL with generated bucket lists and cross join + outer join: https://github.com/tanelpoder/0xtools/blob/master/tools/sql/...
hawkfish-rmgw•4mo ago
I think the real solution is to add support for non-aggregate window functions in the catalogue. Right now, they are all hard-coded, but if they were extensible, there are a number applications in addition to filling that could be supported.
geysersam•4mo ago
From there you can go crazy and define whatever insane interpolations you want.
One limitation is that you can't distinguish the data boundaries from missing values (nulls at the edges vs. nulls in the bulk).
I guess another limitation is if you have multiple nulls after each other...