It feels like a lot of churn and not a lot of measurements or engineering.
I know snowflake does this, they do the query optimization before sending it for execution. So it knows about how many resources will be needed (it can obviously be wrong).
But it seems to have gotten a re-skin recently, and I preferred the previous more pixelated version. Also, there used to be a way to render a "normal" version of the website, but I can't find it anymore.
Why did they write this post if they had not even allowed things to settle. It sounds like someone wanted to claim victory (and promo?)
It is not to serve high quality and timely data to make good decisions, is to avoid hiring a specific role, lol. So bad.
They switched from ClickHouse to DuckDB five months ago, then switched from DuckDB to their own Trino derivative?
The article then ends before saying if the new setup solved their problems.
It’s great that we can now make these migrations with little downtime (I assume?) and try different things, but I wish we could have read more about what testing they did before jumping. The fact that they jumped to DuckDB without realizing it wasn’t going to work at their scale and spent only a few months with it before jumping again is not convincing.
I hoe there’s more to this story that just didn’t make it into the blog posts. As written it’s giving me flashbacks to some teams I worked with who changed tech stacks every six months because they liked migrating more than tuning and maintaining.
> Customers ran more complex queries than we expected
Did the queries change after DuckDB was introduced? Did they do a study of query patterns that could have anticipated this ahead of time?
> Predicting CPU and memory for a specific query was even harder.
What attempts / methodologies were used to try this? What was hard? What failed?
> We'd rather have contributed to the existing community and helped it grow. But we couldn't land changes as fast as we needed, so we ended up carrying patches in a fork
This isn't a BAD approach per se. IME it's common (and necessary) for the large scale users of a platform to maintain a fork for their needs and work with the community and the maintainers to upstream changes when/if the community is ready to accept. That's called influencing. On the other hand, if you toss of a few slop grenades to the community and they reject you shouldn't be surprised. (Not suggesting that's what happening here but the defeatest conclusion may imply it?)
> Missing reconnects and managing TTLs and idle connections at scale is hard
Wonder if they tried pgdog?
Who did the growth estimation of the data? This looks like an example of bad planning.
thisisauserid•50m ago