Cdb: Add support for cdb64

https://cdb.cr.yp.to/download.html

75•kreco•4h ago

Comments

Bolwin•3h ago

Interesting, never heard of this before. I'm assuming the use case is when your data is too large to conveniently fit into memory?

dsr_•3h ago

It is a database for when you read a lot and don't write too often; when a write might be pretty big but not frequent; when you don't want to write a database engine yourself (I.e. figure out what to write and when). And, especially, when corrupting the data would be a big problem.

And it is especially good on copy-on-write filesystems, because it is CoW itself.

bloppe•3h ago

So it's not constant?

tptacek•3h ago

The lookups are ~O(1).

renewiltord•2h ago

Nothing is truly constant lookup in number of elements in nature because we can’t pack it tighter than a sphere.

tptacek•3h ago

It's a database for strictly exact-match lookups for very read-intensive workloads; think systems where the database only changes when the configuration changes, like email alias or domain lookups. It's very simple (a first-level hash table chaining to a second-level open-addressed hash table) and easy to get your head around, but also very limiting; an otherwise strict K-V system that uses b-trees instead of hash tables can do range queries, which you can build a lot of other stuff out of.

Most people would use Redis or SQLite today for what CDB was intended for; CDB will be faster, but for a lot of applications that speed improvement will be sub-threshold for users.

paws•2h ago

For me this answer was helpful and succinct, thank you.

kimos•1h ago

Great reply.

What comes to mind from my experience is storing full shipping rate tables for multiple shipping providers. Those change extremely rarely but are a high throughput exact lookup in a critical path (a checkout).

But we just implemented them in SQLite and deployed that file with the application. Simple clean, effective, and fast. Maybe shipping rate data is smaller than this is intended for, but I doubt using this instead would see a consequential perf increase. Seems niche, like the domain name lookup example.

tptacek•3h ago

Haven't there been 64-bit ports of CDB for ages?

wolfgang42•2h ago

Yes, the modifications you need to support it are trivially obvious (literally just replace “4 bytes” with “8 bytes” everywhere in the spec) and have been implemented by a number of authors, some of which this page links to. I guess it’s nice that they’ve been “officially” acknowledged, though.

gnabgib•3h ago

Title: cdb: Intro (please use the original title) https://news.ycombinator.com/newsguidelines.html

gchamonlive•3h ago

It'd never have crawled out of the new page with that title.

trvz•3h ago

No, the title is much better as it is.

gjvc•1h ago

weak

stevefan1999•3h ago

Can't this be implemented as a PHF: https://en.wikipedia.org/wiki/Perfect_hash_function

wolfgang42•2h ago

CDB is an interesting format, optimized for read-heavy write-rarely[1] random lookups on slow media. This isn’t a very common requirement these days, but it’s convenient for very specific use cases.

[1] You “update” by overwriting the entire file. This is remarkably fast and means that there’s no overhead/tracking for empty space, but it does mean you probably want this to be a fairly rare operation.

I rolled my own cdb reader library for a project a few years ago, and wrote up my notes on the format and its internals here: https://search.feep.dev/blog/post/2022-12-03-cdb-file-format

a-dub•2h ago

GALACTIC SCALE QMAIL that can run efficiently on a 486 AND survive a supernova!

tombert•2h ago

I'm kind of surprised I hadn't heard of this, I could see this being something useful for a few projects. Historically for things in this space I've used RocksDB but RocksDB has given me headaches with unpredictable memory usage for large data sets.

binary132•1h ago

Now I’m curious about working around the writer limitations….

tptacek•1h ago

It's designed to rebuild the whole database with every write, and the format reflects that.

waynesonfire•22m ago

cdb is a fun format to implement! highly recommend it.

Power over Ethernet (PoE) basics and beyond

rlsw – Raylib software OpenGL renderer in less than 5k LOC

OpenBSD 7.8 Released

Replacing a $3000/mo Heroku bill with a $55/mo server

LLMs can get "brain rot"

Show HN: Modshim – a new alternative to monkey-patching in Python

Neural audio codecs: how to get audio into LLMs

The Hidden Engineering of Niagara Falls

Ask HN: Our AWS account got compromised after their outage

Wikipedia says traffic is falling due to AI search summaries and social video

NASA chief suggests SpaceX may be booted from moon mission

ChatGPT Atlas

Mathematicians have found a hidden 'reset button' for undoing rotation

Rectal oxygen delivery might soon be a real medical treatment

Show HN: AutoLearn Skills for self-improving agents

Getting DeepSeek-OCR working on an Nvidia Spark via brute force with Claude Code

Build your own database

60k kids have avoided peanut allergies due to 2015 advice, study finds

Minds, brains, and programs (1980) [pdf]

Understanding conflict resolution and avoidance in PostgreSQL: a complete guide

Doomsday scoreboard

What do we do if SETI is successful?

Foreign hackers breached a US nuclear weapons plant via SharePoint flaws

Flexport Is Hiring SDRs in Chicago

The death of thread per core

Cdb: Add support for cdb64

Show HN: Katakate – Dozens of VMs per node for safe code exec

Erowid - Documenting the Complex Relationship Between Humans and Psychoactives

KDE Connect: Enabling communication between all your devices

Diamond Thermal Conductivity: A New Era in Chip Cooling

Power over Ethernet (PoE) basics and beyond

rlsw – Raylib software OpenGL renderer in less than 5k LOC

OpenBSD 7.8 Released

Replacing a $3000/mo Heroku bill with a $55/mo server

LLMs can get "brain rot"

Show HN: Modshim – a new alternative to monkey-patching in Python

Neural audio codecs: how to get audio into LLMs

The Hidden Engineering of Niagara Falls

Ask HN: Our AWS account got compromised after their outage

Wikipedia says traffic is falling due to AI search summaries and social video

NASA chief suggests SpaceX may be booted from moon mission

ChatGPT Atlas

Mathematicians have found a hidden 'reset button' for undoing rotation

Rectal oxygen delivery might soon be a real medical treatment

Show HN: AutoLearn Skills for self-improving agents

Getting DeepSeek-OCR working on an Nvidia Spark via brute force with Claude Code

Build your own database

60k kids have avoided peanut allergies due to 2015 advice, study finds

Minds, brains, and programs (1980) [pdf]

Understanding conflict resolution and avoidance in PostgreSQL: a complete guide

Doomsday scoreboard

What do we do if SETI is successful?

Foreign hackers breached a US nuclear weapons plant via SharePoint flaws

Flexport Is Hiring SDRs in Chicago

The death of thread per core

Cdb: Add support for cdb64

Show HN: Katakate – Dozens of VMs per node for safe code exec

Erowid - Documenting the Complex Relationship Between Humans and Psychoactives

KDE Connect: Enabling communication between all your devices

Diamond Thermal Conductivity: A New Era in Chip Cooling

Cdb: Add support for cdb64

Comments