Oracle is not imune to software issues. In fact, this year I lost two weekends because of a buggy upgrade on the cloud that left my production cluster in a failed state.

When the author is talking about rolling back his changes, it's not referring to a database, but a version of his library. If someone tried used his new version, I assume the only thing that would have gone wrong is that their code wouldn't work because Pandas didn't support the format.

This article is about how a new version of the Parquet format hasn't been widely adopted, and so now the Parquer community is in a split state where different forces are pulling the direction of the format in two directions, and this happens to be caused by two different areas of focus that don't need to be tightly coupled together.

I don't see how the problems the article discusses relate to the reliability of software.

If a (major) software update cause you an outage, you shouldn’t blame the software, but insufficient testing and validation. Large companies (I worked for many) are slow to adopt new technologies precisely because they are extremely cautious and want to make sure everything was properly tested before they roll it out. That’s also why they still use Oracle and SQL Server (and HP-UX, and IBMi) - these products are working and have been working for generations of employees. The grass needs to be significantly greener for them to consider the move to the other side of their fence.

- https://voltrondata.com/codex/a-new-frontier (links out to others) - https://wesmckinney.com/blog/looking-back-15-years/

in short you can think of a DB as at least 3 decoupled subsystems: UI, compute (query engine), storage. DuckDB has a query engine and storage format, and several UIs (SQL, Python, etc.). Trino is only a query engine (and UIs, everything has UIs). Polars has a query engine. DataFusion is a query engine (and other things). Spark is a query engine. pandas has a query engine

typically query engines are tightly coupled with the overall “product”, but increasingly compute, data (and even more recently via DuckLake metadata), and UI are decoupled allowing you to mix and match parts for a “decomposed database” architecture

quick disclaimer: I worked at Voltron Data but it’s a dead company walking, not trying to advertise for them by any means but the doc I linked is very well written with good information IMO

I haven't though much about it, but I believe the ideal reference implementation would be a highly optimized "service like" process that you run alongside your engine using arrow to share zero copy buffers between the engine and the parquet service. Parquet predates arrow by quite a few years and java was (unfortunately) the standard for big data stuff back then, so they simply stuck with it.

> The way bit-packing has been implemented is to generate 74,000 lines of Java to read/write every combination of bitwidth, endianness and value-length

I think they did this to avoid the dynamic dispatch nature of java. If using C++ or Rust something very similar would happen, but at the compiler level which is a much saner way of doing this kind of thing.

Jesus Christ this isn't 2005 anymore and people need to learn to use the real power of the JVM. It's stuff like this that sets it apart