Open sourcing Dicer: Databricks's auto-sharder

https://www.databricks.com/blog/open-sourcing-dicer-databricks-auto-sharder

85•vivek-jain•3w ago

Comments

ayf•3w ago

Does anyone else have something similar?

What are some use cases that you found are useful?

vivek-jain•3w ago

Sharded in-memory caching turns out to be rather useful at scale :)

Some of the key examples highlighted on our blog are Unity Catalog, which is essentially the metadata layer for Databricks, our Query Orchestration Engine, and our distributed remote cache. See the blog post for more!

louis-paul•3w ago

Sounds related to Google Slicer: https://research.google/pubs/slicer-auto-sharding-for-datace...

atuladya•3w ago

It is similar to Slicer in terms of the abstraction (I built Slicer at Google) but the architecture, implementation and algorithms have a lot of differences

bigwheels•3w ago

Did you also work on this databricks dicery?

hiyer•3w ago

Yes he did. I attended a talk from him on the same, so that's how I know.

WookieRushing•3w ago

These show up once you have a certain scale where it is either cost inefficient or the hot spots are very dynamic. They also try to avoid latency by being eventually consistent sidecars instead of proxies.

I’ve seen them used for traffic routing, storage system metadata systems, distributed cache etc

khaki54•3w ago

Seems weird to call it sharding since it's not sharding indexed datasets or anything like that. Is this just a tool to mitigate Databricks’ internal service-scaling challenges?

atuladya•3w ago

Right - this is not about sharding data/datasets. This is for sharding in-memory state that a service might have. The problem of building services at low cost, high scale, low latency and high throughput is common in many environments including our services at Databricks, and Dicer helps with that.

charleshn•3w ago

> Application pods learn the current assignment through a library called the Slicelet (S for server side). The Slicelet maintains a local cache of the latest assignment by fetching it from the Dicer service and watching for updates. When it receives an updated assignment, the Slicelet notifies the application via a listener API.

For a critical control plane component like this, I tend to prefer a constant work pattern [0], to avoid metastable failures [1], e.g. periodically pull the data instead of relying on notifications.

[0] https://aws.amazon.com/builders-library/reliability-and-cons...

[1] https://brooker.co.za/blog/2021/05/24/metastable.html

jdellithorpe•3w ago

The Dicer Slicelet supports such a pattern, you can poll the assignment directly on the Slicelet:

https://github.com/databricks/dicer/blob/master/dicer/extern...

(btw the notification mechanism itself does not deliver the assignment to the application, only notifies the application that the assignment has changed: https://github.com/databricks/dicer/blob/master/dicer/extern...)

Logic Puzzles: Why the Liar Is the Helpful One

Optical Combs Help Radio Telescopes Work Together

Show HN: Myanon – fast, deterministic MySQL dump anonymizer

The Tao of Programming

Forcing Rust: How Big Tech Lobbied the Government into a Language Mandate

PanelBench: We evaluated Cursor's Visual Editor on 89 test cases. 43 fail

Can You Draw Every Flag in PowerPoint? (Part 2) [video]

Show HN: MCP-baepsae – MCP server for iOS Simulator automation

Make Trust Irrelevant: A Gamer's Take on Agentic AI Safety

Show HN: Sem – Semantic diffs and patches for Git

Hello world does not compile

Show HN: ZigZag – A Bubble Tea-Inspired TUI Framework for Zig

Metaphor+Metonymy: "To love that well which thou must leave ere long"(Sonnet73)

Show HN: Django N+1 Queries Checker

Emacs-tramp-RPC: High-performance TRAMP back end using JSON-RPC instead of shell

Protocol Validation with Affine MPST in Rust

Female Asian Elephant Calf Born at the Smithsonian National Zoo

Show HN: Zest – A hands-on simulator for Staff+ system design scenarios

Show HN: DeSync – Decentralized Economic Realm with Blockchain-Based Governance

Automatic Programming Returns

Why Are There Still So Many Jobs? The History and Future of Workplace Automation [pdf]

The Search Engine Map

Show HN: Souls.directory – SOUL.md templates for AI agent personalities

Real-Time ETL for Enterprise-Grade Data Integration

Economics Puzzle Leads to a New Understanding of a Fundamental Law of Physics

Switzerland's Extraordinary Medieval Library

A new comet was just discovered. Will it be visible in broad daylight?

ESR: Comes the news that Anthropic has vibecoded a C compiler

Frisco residents divided over H-1B visas, 'Indian takeover' at council meeting

If CNN Covered Star Wars