frontpage.
newsnewestaskshowjobs

Made with ♥ by @iamnishanth

Open Source @Github

Open in hackernews

Show HN: ToplingDB - A Persistent Key-Value Store for External Storage

https://github.com/topling/toplingdb
51•rockeetterark•9h ago
As the creator of TerarkDB (acquired by ByteDance in 2019), I have developed ToplingDB in recent years.

ToplingDB is forked from RocksDB, where we have replaced almost all components with more efficient alternatives(db_bench shows ToplingDB is about ~8x faster than RocksDB):

* MemTable: SkipList is replaced by CSPP(Crash Safe Parallel Patricia trie), which is 8x faster.

* SST: BlockBasedTable is replaced by ToplingZipTable, implemented by searchable compression algo, it is very small and fast, typically less than 1μs per lookup:

  * Keys/Indexes are compressed   using NestLoudsTrie(a multi-layer nesting LOUDS succinct trie).

  * Values in a SST are compressed   together with better zip ratio than zstd, and can unzip by a single value at 1GB/sec.

  * BlockCache is no longer needed, double caching(BlockCache & PageCache) is avoided
Other hotspots are also improved:

* Flush MemTable to L0 is omited, greatly reducing write amp and is very friendly for large(GB) MemTable

  * MemTable   serves as the index of Key to "value position in WAL log"

  * Since WAL file content almost always in page cache, thus value content can be efficiently accessed by mmap

  * When Flush happens, MemTable is dumpped as an SST and WAL is treated as a blob file

    * CSPP MemTable use integer index instead of physical pointers, thus in-memory format is exactly same with in-file format
* Prefix cache for searching candidate SSTs and prefix cache for scanning by iterators

  * Caching fixed len key prefix into an array, binary search it as an uint array
* Distributed compaction(superior replacement to rocksdb remote compaction)

  * Gracefully support MergeOperator, CompactionFilter, PropertiesCollector...

  * Out of the box, development efforts are significantly reduced

  * Very easy to share compaction service on spot instances for many DB nodes
Useful Bonus Feature:

* Config by json/yaml: can config almost all features

* Optional embeded WebView: show db structures in web browser, refreshing pages like animation

* Online update db configs by http

MySQL integration, ToplingDB has integrated into MySQL by MyTopling, which is forked from MyRocks with great improvements, like improvements of ToplingDB on RocksDB:

* WBWI(WriteBatchWithIndex): like MemTable, SkipList is replace with CSPP, 20x faster(speedup is more than MemTable).

* LockManager & LockTracker: 10x faster

* Encoding & Decoding: 5x faster

* Others ....

MyRocks has many disadvantages compared to InnoDB, while MyTopling outperforms InnoDB at almost all aspect - excluding feature differences.

We have create ~100 PRs for RocksDB, in which ~40 were accepted. Our PRs are mostly "small" changes, since big changes are not likely accepted.

ToplingDB has been deployed in numerous production environments.

Welcome every one using ToplingDB & MyTopling, and discuss in https://github.com/topling/toplingdb/discussions

Comments

ChocolateGod•7h ago
I'm confused what makes this cloud native?
dboreham•5h ago
It has an embedded http server?
faizshah•4h ago
From what I gather it has an embedded http control plane, yaml/json config for plugins, prometheus integration, and distributed compaction workers on separate, potentially serverless, hosts.
andybak•6h ago
This is failing my "Can I figure out what the hell it is in 60 seconds?" test.

Sometimes that means I'm just not the target market. I do do web dev (among other things) so that doesn't seem to be the case at first glance?

faizshah•5h ago
It’s RocksDB but faster because data can be searched while still compressed allowing you to load more records in less cache/ram leading to up to 10x performance of RocksDB. It adds an embedded http control plane as well as supporting other extensions like MyRocks (MySQL) and Todis (redis compatibility).

Or at least thats what I got from it correct me if I am wrong rockeet.

alexpadula•5h ago
Very extensive, great work on TerarkDB and Topling!
dangoodmanUT•5h ago
Without better (english) docs it will be hard to get adoption, unfortunately. 8x perf gain over rocksdb is... a lot... unless you're poking at particularly bad metrics.
absoluteunit1•4h ago
For the laymen folks reading this - what are the ideal use cases for this?
nbf_1995•1h ago
Like RocksDB from which this appears to be forked, the primary usage is as a storage engine for other applications/databases. Compared to rocksdb, it seems like ToplingDB has added more facilities to better support distributed use-cases.

Some databases that utilize RocksDB for their storage engine: https://kvrocks.apache.org/ - Redis/ValKey compatible distributed database with disk persistence via RockDB. https://github.com/pingcap/tidb - MySQL compatible distributed database. Mentioned elsewhere in this thread. https://github.com/tikv/tikv - Distributed, transactional, key value store. Originally by the same company as TiDB.

In theory you could use it as an in-process KV store similar to how SQLite provides an in process sql database, but the api is far from ergonomic for that use case.

alex7o•3h ago
What does it have to do with external storage in this context, does it mean S3. Initially I thought it is a db for thumb drives?
ozgrakkurt•3h ago
Would be really interesting to have faster compilation and more simplicity (auto tuning parameters etc.) compared to rocksdb. In my experience rocksdb performance is very good and it is reliable but it is a pain to integrate into the build process and has too many configurations
esafak•3h ago
A distributed KV-store plus a relational layer makes it a competitor to NewSQL databases like TiDB, which is also based on Facebook's RocksDB.

It doesn't look like it's very actively developed: https://github.com/topling/toplingdb/pulse/monthly

To the OP who's developing it: I suggest polishing your README. Provide a simple installation tutorial, maybe a trial offering like tidbcloud.com, and comparative benchmark results, since you advertise your performance.

jauntywundrkind•38m ago
It's quite active. They just aren't using GitHub pull requests in their workflow, which is what GitHub Pulse measures. https://github.com/topling/toplingdb/commits/memtable_as_log...

The Fed says this is a cube of $1M. They're off by half a million

https://calvin.sh/blog/fed-lie/
483•c249709•3h ago•209 comments

The Hoyle State (2021)

https://johncarlosbaez.wordpress.com/2021/02/04/the-hoyle-state/
21•gone35•1h ago•1 comments

Feasibility study of a mission to Sedna - Nuclear propulsion and solar sailing

https://arxiv.org/abs/2506.17732
123•speckx•5h ago•39 comments

What should a native DOM templating API look like?

https://justinfagnani.com/2025/06/30/what-should-a-dom-templating-api-look-like/
32•jdkoeck•1h ago•6 comments

Ask HN: Who is hiring? (July 2025)

112•whoishiring•4h ago•148 comments

Show HN: Spegel, a Terminal Browser That Uses LLMs to Rewrite Webpages

https://simedw.com/2025/06/23/introducing-spegel/
238•simedw•6h ago•111 comments

Experience converting a mathematical software package to C++20 modules [PDF]

https://arxiv.org/abs/2506.21654
67•vblanco•5h ago•9 comments

I built something that changed my friend group's social fabric

https://blog.danpetrolito.xyz/i-built-something-that-changed-my-friend-gro-social-fabric/
421•dandano•3d ago•182 comments

Ask HN: Who wants to be hired? (July 2025)

15•whoishiring•4h ago•84 comments

Show HN: HackerNewt - Breadth-first exploring HN client for iOS

https://apps.apple.com/us/app/hackernewt-for-hacker-news/id6448201970
28•hnand•3h ago•9 comments

Cua (YC X25) Is Hiring a Founding Engineer

https://www.ycombinator.com/companies/cua/jobs/dIskIB1-founding-engineer-cua-yc-x25
1•GreenGames•2h ago

MicroPython on M68k Mac

https://social.afront.org/@stylus/114749858767978151
12•zdw•2h ago•0 comments

When Did Nature Burst into Vivid Color?

https://www.quantamagazine.org/when-did-nature-burst-into-vivid-color-20250627/
52•jandrewrogers•4d ago•28 comments

All Good Editors Are Pirates: In Memory of Lewis H. Lapham

https://www.laphamsquarterly.org/roundtable/all-good-editors-are-pirates
27•Caiero•2d ago•3 comments

Code⇄GUI bidirectional editing via LSP

https://jamesbvaughan.com/bidirectional-editing/
11•jamesbvaughan•2h ago•1 comments

OpenFLOW – Quickly make beautiful infrastructure diagrams local to your machine

https://github.com/stan-smith/OpenFLOW
232•x0z•12h ago•56 comments

Graph Theory Applications in Video Games

https://utk.claranguyen.me/talks.php?id=videogames
26•haywirez•3d ago•1 comments

Show HN: I built the tool I wished existed for moving Stripe between countries

https://www.stripemove.com/
55•felphos•6h ago•29 comments

Swearing as a Response to Pain: Assessing Effects of Novel Swear Words

https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2020.00723/full
4•sega_sai•2d ago•0 comments

Show HN: Jobs by Referral: Find jobs in your LinkedIn network

https://jobsbyreferral.com/
90•nicksergeant•6h ago•50 comments

Aesop in Words of One Syllable

https://blog.pgdp.net/2025/07/01/aesop-in-words-of-one-syllable/
21•sohkamyung•5h ago•15 comments

Exploring Trichromacy through Maxwell's Color Experiment (2023)

https://maxwell.kohterai.com/
30•niwrad•2d ago•5 comments

1KB JavaScript Demoscene Challenge Just Launched

18•babakode•48m ago•0 comments

Amazon Is on the Cusp of Using More Robots Than Humans in Its Warehouses

https://www.wsj.com/tech/amazon-warehouse-robots-automation-942b814f
36•jbredeche•2h ago•32 comments

Sam Altman Slams Meta's AI Talent Poaching: 'Missionaries Will Beat Mercenaries'

https://www.wired.com/story/sam-altman-meta-ai-talent-poaching-spree-leaked-messages/
19•spenvo•1h ago•32 comments

I write type-safe generic data structures in C

https://danielchasehooper.com/posts/typechecked-generic-c-data-structures/
393•todsacerdoti•1d ago•160 comments

Embabel Agent Framework for the JVM

https://github.com/embabel/embabel-agent
29•t0mas88•2d ago•4 comments

PlanetScale for Postgres

https://planetscale.com/blog/planetscale-for-postgres
80•adocomplete•4h ago•8 comments

The new skill in AI is not prompting, it's context engineering

https://www.philschmid.de/context-engineering
829•robotswantdata•22h ago•464 comments

The hidden JTAG in a Qualcomm/Snapdragon device’s USB port

https://www.linaro.org/blog/hidden-jtag-qualcomm-snapdragon-usb/
228•denysvitali•1d ago•33 comments