frontpage.
newsnewestaskshowjobs

Made with ♥ by @iamnishanth

Open Source @Github

Open in hackernews

Show HN: ToplingDB - A Persistent Key-Value Store for External Storage

https://github.com/topling/toplingdb
53•rockeetterark•11h ago
As the creator of TerarkDB (acquired by ByteDance in 2019), I have developed ToplingDB in recent years.

ToplingDB is forked from RocksDB, where we have replaced almost all components with more efficient alternatives(db_bench shows ToplingDB is about ~8x faster than RocksDB):

* MemTable: SkipList is replaced by CSPP(Crash Safe Parallel Patricia trie), which is 8x faster.

* SST: BlockBasedTable is replaced by ToplingZipTable, implemented by searchable compression algo, it is very small and fast, typically less than 1μs per lookup:

  * Keys/Indexes are compressed   using NestLoudsTrie(a multi-layer nesting LOUDS succinct trie).

  * Values in a SST are compressed   together with better zip ratio than zstd, and can unzip by a single value at 1GB/sec.

  * BlockCache is no longer needed, double caching(BlockCache & PageCache) is avoided
Other hotspots are also improved:

* Flush MemTable to L0 is omited, greatly reducing write amp and is very friendly for large(GB) MemTable

  * MemTable   serves as the index of Key to "value position in WAL log"

  * Since WAL file content almost always in page cache, thus value content can be efficiently accessed by mmap

  * When Flush happens, MemTable is dumpped as an SST and WAL is treated as a blob file

    * CSPP MemTable use integer index instead of physical pointers, thus in-memory format is exactly same with in-file format
* Prefix cache for searching candidate SSTs and prefix cache for scanning by iterators

  * Caching fixed len key prefix into an array, binary search it as an uint array
* Distributed compaction(superior replacement to rocksdb remote compaction)

  * Gracefully support MergeOperator, CompactionFilter, PropertiesCollector...

  * Out of the box, development efforts are significantly reduced

  * Very easy to share compaction service on spot instances for many DB nodes
Useful Bonus Feature:

* Config by json/yaml: can config almost all features

* Optional embeded WebView: show db structures in web browser, refreshing pages like animation

* Online update db configs by http

MySQL integration, ToplingDB has integrated into MySQL by MyTopling, which is forked from MyRocks with great improvements, like improvements of ToplingDB on RocksDB:

* WBWI(WriteBatchWithIndex): like MemTable, SkipList is replace with CSPP, 20x faster(speedup is more than MemTable).

* LockManager & LockTracker: 10x faster

* Encoding & Decoding: 5x faster

* Others ....

MyRocks has many disadvantages compared to InnoDB, while MyTopling outperforms InnoDB at almost all aspect - excluding feature differences.

We have create ~100 PRs for RocksDB, in which ~40 were accepted. Our PRs are mostly "small" changes, since big changes are not likely accepted.

ToplingDB has been deployed in numerous production environments.

Welcome every one using ToplingDB & MyTopling, and discuss in https://github.com/topling/toplingdb/discussions

Comments

ChocolateGod•9h ago
I'm confused what makes this cloud native?
dboreham•7h ago
It has an embedded http server?
faizshah•6h ago
From what I gather it has an embedded http control plane, yaml/json config for plugins, prometheus integration, and distributed compaction workers on separate, potentially serverless, hosts.
andybak•8h ago
This is failing my "Can I figure out what the hell it is in 60 seconds?" test.

Sometimes that means I'm just not the target market. I do do web dev (among other things) so that doesn't seem to be the case at first glance?

faizshah•8h ago
It’s RocksDB but faster because data can be searched while still compressed allowing you to load more records in less cache/ram leading to up to 10x performance of RocksDB. It adds an embedded http control plane as well as supporting other extensions like MyRocks (MySQL) and Todis (redis compatibility).

Or at least thats what I got from it correct me if I am wrong rockeet.

alexpadula•8h ago
Very extensive, great work on TerarkDB and Topling!
dangoodmanUT•7h ago
Without better (english) docs it will be hard to get adoption, unfortunately. 8x perf gain over rocksdb is... a lot... unless you're poking at particularly bad metrics.
absoluteunit1•7h ago
For the laymen folks reading this - what are the ideal use cases for this?
nbf_1995•3h ago
Like RocksDB from which this appears to be forked, the primary usage is as a storage engine for other applications/databases. Compared to rocksdb, it seems like ToplingDB has added more facilities to better support distributed use-cases.

Some databases that utilize RocksDB for their storage engine: https://kvrocks.apache.org/ - Redis/ValKey compatible distributed database with disk persistence via RockDB. https://github.com/pingcap/tidb - MySQL compatible distributed database. Mentioned elsewhere in this thread. https://github.com/tikv/tikv - Distributed, transactional, key value store. Originally by the same company as TiDB.

In theory you could use it as an in-process KV store similar to how SQLite provides an in process sql database, but the api is far from ergonomic for that use case.

alex7o•5h ago
What does it have to do with external storage in this context, does it mean S3. Initially I thought it is a db for thumb drives?
ozgrakkurt•5h ago
Would be really interesting to have faster compilation and more simplicity (auto tuning parameters etc.) compared to rocksdb. In my experience rocksdb performance is very good and it is reliable but it is a pain to integrate into the build process and has too many configurations
esafak•5h ago
A distributed KV-store plus a relational layer makes it a competitor to NewSQL databases like TiDB, which is also based on Facebook's RocksDB.

It doesn't look like it's very actively developed: https://github.com/topling/toplingdb/pulse/monthly

To the OP who's developing it: I suggest polishing your README. Provide a simple installation tutorial, maybe a trial offering like tidbcloud.com, and comparative benchmark results, since you advertise your performance.

jauntywundrkind•3h ago
It's quite active. They just aren't using GitHub pull requests in their workflow, which is what GitHub Pulse measures. https://github.com/topling/toplingdb/commits/memtable_as_log...

Show HN: Spegel, a Terminal Browser That Uses LLMs to Rewrite Webpages

https://simedw.com/2025/06/23/introducing-spegel/
281•simedw•9h ago•131 comments

Show HN: Core – open source memory graph for LLMs – shareable, user owned

https://github.com/RedPlanetHQ/core
34•Manik_agg•5h ago•3 comments

Show HN: HackerNewt - Breadth-first exploring HN client for iOS

https://apps.apple.com/us/app/hackernewt-for-hacker-news/id6448201970
34•hnand•5h ago•22 comments

Show HN: Jobs by Referral: Find jobs in your LinkedIn network

https://jobsbyreferral.com/
100•nicksergeant•9h ago•53 comments

Show HN: Arch-Router – 1.5B model for LLM routing by preferences, not benchmarks

33•adilhafeez•4h ago•8 comments

Show HN: I built the tool I wished existed for moving Stripe between countries

https://www.stripemove.com/
71•felphos•9h ago•36 comments

Show HN: Generate presentation slides from Jira backlog. + Demo-playground

https://agileplus.io/
2•anthonyag•1h ago•0 comments

Show HN: A continuation of IRS Direct File that can be self-hosted

https://github.com/openfiletax/openfile
194•elijahwright_•23h ago•23 comments

Show HN: ToplingDB - A Persistent Key-Value Store for External Storage

https://github.com/topling/toplingdb
53•rockeetterark•11h ago•13 comments

Show HN: Runik – Turn fan wikis into e-reader dictionaries

https://github.com/Runik-3/core
2•skoutXII•3h ago•1 comments

Show HN: Exceptional free templates, component libraries and boilerplates

https://htmlrev.com/
13•devluc•10h ago•5 comments

Show HN: Fixstars AIBooster – Accelerate AI Training and Cut GPU Costs

https://www.fixstars.com/en/ai/ai-booster
4•aki_asahara•3h ago•0 comments

Show HN: A tool to minimize echo chambers on social media

https://www.tandfonline.com/doi/full/10.1080/29974100.2025.2517191
3•academic_84572•5h ago•0 comments

Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

https://github.com/M4THYOU/TokenDagger
270•matthewolfe•1d ago•70 comments

Show HN: Open-Source International Space Station Tracker ESP32/Arduino for $20

https://github.com/GuitarML/SpaceStationTracker
67•keyth72•4d ago•18 comments

Show HN: New Ensō – first public beta

https://untested.sonnet.io/notes/new-enso-first-public-beta/
241•rpastuszak•1d ago•84 comments

Show HN: We're two coffee nerds who built an AI app to track beans and recipes

https://beanbook.app
52•rokeyzhang•1d ago•51 comments

Show HN: Local LLM Notepad – run a GPT-style model from a USB stick

https://github.com/runzhouye/Local_LLM_Notepad
28•davidye324•22h ago•6 comments

Show HN: Lifp – A Lisp Built on Bun

https://github.com/shikaan/lifp
3•shikaan•7h ago•0 comments

Show HN: Get off your a** and do push-ups

https://timm.so/features/push-up-to-unlock
8•itsharveenatwal•7h ago•6 comments

Show HN: Subwise – A Smart Subscription Tracker with Browser Integration

https://www.usesubwise.com/
2•paulsabandal•8h ago•0 comments

Show HN: Octelium – FOSS Alternative to Teleport, Cloudflare, Tailscale, Ngrok

https://github.com/octelium/octelium
343•geoctl•2d ago•147 comments

Show HN: I'm an airline pilot – I built interactive graphs/globes of my flights

https://jameshard.ing/pilot
1519•jamesharding•4d ago•196 comments

Show HN: Private real-time dictation app for Mac

https://github.com/aviaryan/Very-Fast-Dictation
17•aviaryan•1d ago•8 comments

Show HN: I built a daily sunlight tracker

https://www.lumehealth.io/products
42•vickipow•4d ago•32 comments

Show HN: C.O.R.E – Opensource, user owned, shareable memory for Claude, Cursor

https://github.com/RedPlanetHQ/core
11•Manik_agg•1d ago•4 comments

Show HN: A cross-platform terminal emulator written in Java

https://github.com/sebkur/forceterm
3•sebkur•14h ago•0 comments

Show HN: SpeechAnalyzerDylib – C-FFI for macOS 26.0 SpeechAnalyzer API

https://github.com/aethiopicuschan/speech-analyzer-dylib
2•aethiopicuschan•15h ago•0 comments

Show HN: I Built a Paul Graham AI Advisor for Founders and Hackers

https://paulgraham.resurrect.space
18•vednig•13h ago•19 comments

Show HN: Summle – A little maths Game

https://summle.net
26•kirchhoff•5d ago•10 comments