For every other data engineer or someone in higher hierarchy down the road comes to a choiuce of Apache Iceberg or Databricks Delta Lake, so we went ahead and benchmarked both systems. Just sharing our experience here.
TL;DR
Both formats have their perks: Apache Iceberg offers an open, flexible architecture with surprisingly fast query performance in some cases, while Databricks Delta Lake provides a tightly managed, all-in-one experience where most of the operational overhead is handled for you.
Setup & Methodology
We used the TPC-H 1 TB dataset which is a dataset of about 8.66 billion rows across 8 tables to compare the two stacks end-to-end: ingestion and analytics.
For the Iceberg setup:
We ingested data from PostgreSQL into Apache Iceberg tables on S3, orchestrated through OLake’s high-throughput CDC pipeline using AWS Glue as catalog and EMR Spark for query..
Ingestion used 32 parallel threads with chunked, resumable snapshots, ensuring high throughput.
On the query side, we tuned Spark similarly to Databricks (raised shuffle partitions to 128 and disabled vectorised reads due to Arrow buffer issues).
For the Databricks Delta Lake setup:
Data was loaded via the JDBC connector from PostgreSQL into Delta tables in 200k-row batches. Databricks’ managed runtime automatically applied file compaction and optimized writes.
Queries were run using the same 22 TPC-H analytics queries for a fair comparison.
This setup made sure we were comparing both ingestion performance and analytical query performance under realistic, production-style workloads.
What We Found
We used OLake to ingest to Iceberg and was about 2x faster - 12 hours vs 25.7 hours on Databricks thanks to parallel chunked ingestion.
Iceberg ran the full TPC-H suite 18% faster than Databricks.
Cost: Infra cost was 61% lower on Iceberg + OLake (around $21.95 vs $50.71 for the same run).
here are the overall result and our ideology on this-
Databricks still wins on ease-of-use: you just click and go. Cluster setup, Spark tuning, and governance are all handled automatically. That’s great for teams that want a managed ecosystem and don’t want to deal with infrastructure.
But if your team is comfortable managing a Glue/AWS stack and handling a bit more complexity, Iceberg + OLake’s open architecture wins on pure numbers faster at scale, lower cost, and full engine flexibility (Spark, Trino, Flink) without vendor lock-in.
read our article to know more on our steps followed and the overall benchmarks and the numbers around it curious to know what you people think ofcourse these are numbers but it largely depends on your experience too of how you adopted in your org
techandtry•2mo ago
Interesting write-up. What I’ve seen in practice is that benchmarks like these usually highlight the best-case behavior of Iceberg vs the default behavior of Delta. The real question ends up being how much manual tuning a team is willing to take on. Iceberg can absolutely outperform Delta if you’re intentional about ingestion parallelism or eve catalog setup.
But I’m also curious about the long-term operational side - how does the maintenance overhead evolve after a few months of production load?that maybe someone that have transitioned can answer
Cappybara12•2mo ago
TL;DR Both formats have their perks: Apache Iceberg offers an open, flexible architecture with surprisingly fast query performance in some cases, while Databricks Delta Lake provides a tightly managed, all-in-one experience where most of the operational overhead is handled for you.
Setup & Methodology
We used the TPC-H 1 TB dataset which is a dataset of about 8.66 billion rows across 8 tables to compare the two stacks end-to-end: ingestion and analytics.
For the Iceberg setup:
We ingested data from PostgreSQL into Apache Iceberg tables on S3, orchestrated through OLake’s high-throughput CDC pipeline using AWS Glue as catalog and EMR Spark for query.. Ingestion used 32 parallel threads with chunked, resumable snapshots, ensuring high throughput. On the query side, we tuned Spark similarly to Databricks (raised shuffle partitions to 128 and disabled vectorised reads due to Arrow buffer issues).
For the Databricks Delta Lake setup: Data was loaded via the JDBC connector from PostgreSQL into Delta tables in 200k-row batches. Databricks’ managed runtime automatically applied file compaction and optimized writes. Queries were run using the same 22 TPC-H analytics queries for a fair comparison.
This setup made sure we were comparing both ingestion performance and analytical query performance under realistic, production-style workloads.
What We Found
We used OLake to ingest to Iceberg and was about 2x faster - 12 hours vs 25.7 hours on Databricks thanks to parallel chunked ingestion.
Iceberg ran the full TPC-H suite 18% faster than Databricks.
Cost: Infra cost was 61% lower on Iceberg + OLake (around $21.95 vs $50.71 for the same run).
here are the overall result and our ideology on this-
Databricks still wins on ease-of-use: you just click and go. Cluster setup, Spark tuning, and governance are all handled automatically. That’s great for teams that want a managed ecosystem and don’t want to deal with infrastructure.
But if your team is comfortable managing a Glue/AWS stack and handling a bit more complexity, Iceberg + OLake’s open architecture wins on pure numbers faster at scale, lower cost, and full engine flexibility (Spark, Trino, Flink) without vendor lock-in.
read our article to know more on our steps followed and the overall benchmarks and the numbers around it curious to know what you people think ofcourse these are numbers but it largely depends on your experience too of how you adopted in your org