frontpage.
newsnewestaskshowjobs

Made with ♥ by @iamnishanth

Open Source @Github

Open in hackernews

Show HN: ToplingDB - A Persistent Key-Value Store for External Storage

https://github.com/topling/toplingdb
28•rockeetterark•5h ago
As the creator of TerarkDB (acquired by ByteDance in 2019), I have developed ToplingDB in recent years.

ToplingDB is forked from RocksDB, where we have replaced almost all components with more efficient alternatives(db_bench shows ToplingDB is about ~8x faster than RocksDB):

* MemTable: SkipList is replaced by CSPP(Crash Safe Parallel Patricia trie), which is 8x faster.

* SST: BlockBasedTable is replaced by ToplingZipTable, implemented by searchable compression algo, it is very small and fast, typically less than 1μs per lookup:

  * Keys/Indexes are compressed   using NestLoudsTrie(a multi-layer nesting LOUDS succinct trie).

  * Values in a SST are compressed   together with better zip ratio than zstd, and can unzip by a single value at 1GB/sec.

  * BlockCache is no longer needed, double caching(BlockCache & PageCache) is avoided
Other hotspots are also improved:

* Flush MemTable to L0 is omited, greatly reducing write amp and is very friendly for large(GB) MemTable

  * MemTable   serves as the index of Key to "value position in WAL log"

  * Since WAL file content almost always in page cache, thus value content can be efficiently accessed by mmap

  * When Flush happens, MemTable is dumpped as an SST and WAL is treated as a blob file

    * CSPP MemTable use integer index instead of physical pointers, thus in-memory format is exactly same with in-file format
* Prefix cache for searching candidate SSTs and prefix cache for scanning by iterators

  * Caching fixed len key prefix into an array, binary search it as an uint array
* Distributed compaction(superior replacement to rocksdb remote compaction)

  * Gracefully support MergeOperator, CompactionFilter, PropertiesCollector...

  * Out of the box, development efforts are significantly reduced

  * Very easy to share compaction service on spot instances for many DB nodes
Useful Bonus Feature:

* Config by json/yaml: can config almost all features

* Optional embeded WebView: show db structures in web browser, refreshing pages like animation

* Online update db configs by http

MySQL integration, ToplingDB has integrated into MySQL by MyTopling, which is forked from MyRocks with great improvements, like improvements of ToplingDB on RocksDB:

* WBWI(WriteBatchWithIndex): like MemTable, SkipList is replace with CSPP, 20x faster(speedup is more than MemTable).

* LockManager & LockTracker: 10x faster

* Encoding & Decoding: 5x faster

* Others ....

MyRocks has many disadvantages compared to InnoDB, while MyTopling outperforms InnoDB at almost all aspect - excluding feature differences.

We have create ~100 PRs for RocksDB, in which ~40 were accepted. Our PRs are mostly "small" changes, since big changes are not likely accepted.

ToplingDB has been deployed in numerous production environments.

Welcome every one using ToplingDB & MyTopling, and discuss in https://github.com/topling/toplingdb/discussions

Comments

ChocolateGod•2h ago
I'm confused what makes this cloud native?
dboreham•1h ago
It has an embedded http server?
andybak•1h ago
This is failing my "Can I figure out what the hell it is in 60 seconds?" test.

Sometimes that means I'm just not the target market. I do do web dev (among other things) so that doesn't seem to be the case at first glance?

faizshah•1h ago
It’s RocksDB but faster because data can be searched while still compressed allowing you to load more records in less cache/ram leading to up to 10x performance of RocksDB. It adds an embedded http control plane as well as supporting other extensions like MyRocks (MySQL) and Todis (redis compatibility).

Or at least thats what I got from it correct me if I am wrong rockeet.

alexpadula•1h ago
Very extensive, great work on TerarkDB and Topling!
dangoodmanUT•1h ago
Without better (english) docs it will be hard to get adoption, unfortunately. 8x perf gain over rocksdb is... a lot... unless you're poking at particularly bad metrics.
absoluteunit1•30m ago
For the laymen folks reading this - what are the ideal use cases for this?
b0a04gl•11m ago
terarkdb team's second attempt after bytedance acquired their first project. they forked rocksdb and rebuilt the core components - skip lists became cspp tries, new sst format, distributed compaction added on top. the upstream tension is clear from their stats: 100 prs submitted to rocksdb, only 40 accepted. rocksdb maintainers have to be conservative about breaking changes, so the team went with their own fork instead.

you claim crash-safe patricia tries, but how do you handle partial writes during node splits? what's the recovery protocol if you crash mid-rebalancing? would lve to see the specific wal interactions here.

also if you can throw some light on below parts, thanks: > distributed compaction > compression trade-offs

Do we need a new social media build on generative AI?

1•imwoody•2m ago•1 comments

Has AI made "learn to code" obsolete?

https://www.freethink.com/artificial-intelligence/learn-to-code
1•daviducolo•2m ago•0 comments

ORMs are criticized for the wrong reasons

https://www.getlago.com/blog/orms-are-criticized-for-the-wrong-reasons
1•FinnLobsien•4m ago•0 comments

Show HN: Road Quality App in Oxford, UK, Based on Street View Imagery

https://philippopien.users.earthengine.app/view/oxford-uk-road-quality
1•deepvoltaire•5m ago•0 comments

Ask HN: Who is hiring? (July 2025)

4•whoishiring•6m ago•10 comments

Ask HN: Freelancer? Seeking freelancer? (July 2025)

1•whoishiring•6m ago•9 comments

Ask HN: Who wants to be hired? (July 2025)

2•whoishiring•6m ago•9 comments

MotherDuck launches managed DuckLake service

https://motherduck.com/blog/announcing-ducklake-support-motherduck-preview/
1•ryguyrg•6m ago•2 comments

I Shipped a macOS App Built by Claude Code

https://www.indragie.com/blog/i-shipped-a-macos-app-built-entirely-by-claude-code
1•indragie•6m ago•0 comments

Show HN: Framework for LLM Iterative Refinement Until Mathematical Convergence

https://github.com/hankbesser/recursive-companion
2•henrybe473•6m ago•0 comments

Show HN: FocusTrain – Chrome Extension for "Tab Hoppers"

https://focustrain.app/
1•bhavaniravi•7m ago•0 comments

Microsoft Says New AI Diagnosed Patients 4 Times More Accurately Than Doctors

https://www.wired.com/story/microsoft-medical-superintelligence-diagnosis/
1•BeetleB•8m ago•0 comments

Show HN: I built music discovery where songs disappear forever

https://sharevibes.app
1•lucascliberato•9m ago•0 comments

Republicans nix measure that would have blocked states from regulating AI

https://qz.com/ai-moratorium-senate-big-beautiful-bill-stripped
1•amirkabbara•10m ago•0 comments

Search Navigator – Vim-Style Keyboard Navigation for Google Search

https://chromewebstore.google.com/detail/search-navigator/fpinaaaiplppifhmkjdfkimodkkdnoha
1•nwatab•11m ago•1 comments

Pakistani freelancers building cracking websites for stealer-delivery

https://www.intrinsec.com/pakistani-freelancers-building-cracking-websites-for-stealer-delivery/
2•campuscodi•13m ago•0 comments

Superhuman is being acquired by Grammarly to build the future of work

https://blog.superhuman.com/superhuman-is-being-acquired-by-grammarly/
1•Timothee•13m ago•1 comments

Show HN: Flow – A Command-Line Tool for Deep Work

https://github.com/e6a5/flow
4•ihiep•14m ago•2 comments

Artificial heart developer Carmat faces insolvency

https://www.medicaldevice-network.com/news/french-artificial-heart-maker-carmat-files-for-insolvency/
1•Jyaif•14m ago•0 comments

Ekai: AI Twin for Slack Users

https://yourekai.com/
1•jordigh•14m ago•0 comments

Why Apple's Car Failed Before It Even Launched

https://www.carsandhorsepower.com/featured/why-apple-s-car-failed-before-it-even-launched-and-what-it-means-for-tech-giants-in-auto
1•Anumbia•15m ago•1 comments

Scientists discover never-before-seen part of human cells

https://www.livescience.com/health/scientists-discover-never-before-seen-part-of-human-cells-and-it-looks-like-a-snowman-wearing-a-scarf
1•Brajeshwar•15m ago•0 comments

MIT's high-tech 'bubble wrap' turns air into safe drinking water

https://www.livescience.com/technology/engineering/mits-high-tech-bubble-wrap-turns-air-into-safe-drinking-water-even-in-death-valley
1•Brajeshwar•15m ago•0 comments

Long-dead satellite emits strong radio signal, puzzling astronomers

https://www.cnn.com/2025/06/30/science/satellite-strange-radio-signal-relay-2
2•Brajeshwar•16m ago•0 comments

Apple claims San Jose engineer stole trade secrets, took them to rival

https://www.mercurynews.com/2025/06/30/apple-san-jose-engineer-stole-trade-secrets-rival/
1•basisword•16m ago•0 comments

Show HN: Created a script to run commands in an isolated VM for AI tool calling

https://github.com/bigattichouse/scratchpad
2•bigattichouse•18m ago•0 comments

Anthropologist uncovers the 11,000-year history of avocado domestication

https://phys.org/news/2025-06-anthropologist-uncovers-year-history-avocado.html
1•PaulHoule•21m ago•0 comments

Schools serving undocumented kids go underground

https://19thnews.org/2025/07/schools-serving-undocumented-kids-go-underground/
1•benwerd•22m ago•0 comments

What Is InferenceOps

https://www.bentoml.com/blog/what-is-inference-ops
2•sherlockxu•22m ago•0 comments

Why is it so hard for startups to compete with Cadence?

https://www.zach.be/p/why-is-it-so-hard-for-startups-to
1•zachbee•25m ago•0 comments