Replacing Protobuf with Rust to go 5 times faster

https://pgdog.dev/blog/replace-protobuf-with-rust

40•whiteros_e•2h ago

Comments

nottorp•1h ago

Are they sure it's because Rust? Perhaps if they rewrite Protobuf in Rust it will be as slow as the current implementation.

They changed the persistence system completely. Looks like from a generic solution to something specific to what they're carrying across the wire.

They could have done it in Lua and it would have been 3x faster.

consp•1h ago

If they made the headline something on the line of "replacing protobuf with a native, optimized implementation" would not get the same attention as putting rust in the title to attract the everything-in-rust-is-better crowd.

desiderantes•1h ago

That never happens. Instead, it always attracts the opposite group, the Rust complainers, where they go and complain about how "the everything-in-rust-is-better crowd created yet another fake headline to pretend that Rust is the panacea". Which results in a lot of engagement. Old ragebait trick.

misja111•1h ago

Correct, this has very little to do with Rust. But it wouldn't have made the front page without it.

locknitpicker•1h ago

Yes you are absolutely right. The article even outright admits that Rust had nothing to do with it. From the article:

> Protobuf is fast, but not using Protobuf is faster.

The blog post reads like an unserious attempt to repeat a Rust meme.

alias_neo•1h ago

I was equally confused by the headline.

I wonder if it's just poorly worded and they meant to say something like "Replacing Protobuf with some native calls [in Rust]".

embedding-shape•1h ago

It's devbait, not many of us can resist bikeshedding about the title which obviously doesn't accurately reflect the article contents. And the article contents are self-aware enough to admit this to itself too, yet the title remains.

win311fwg•1h ago

The title would suggest that it was already written in Rust; that it was the rewrite in Go that brought five times faster.

IshKebab•1h ago

I vaguely recall that there's a Rust macro to automatically convert recursive functions to iterative.

But I would just increase the stack size limit if it ever becomes a problem. As far as I know the only reason it is so small is because of address space exhaustion which only affects 32-bit systems.

embedding-shape•1h ago

> I vaguely recall that there's a Rust macro to automatically convert recursive functions to iterative.

Isn't that just TCO or similar? Usually a part of the compiler/core of the language itself, AFAIK.

koverstreet•45m ago

I haven't been following become/TCO in Rust - but what I've usually seen is TCO getting flipped off because it interferes with backtraces and debugging.

So I think there's value in providing it as an explicit opt-in; that way when you're reading the code, you know to account for it when you're looking at backtraces.

Additionally, if you're relying on TCO it might be a major bug if the compiler isn't able to apply it - and optimizations that aren't applied are normally invisible. This might mean you could get an error if you're expecting TCO and you or the compiler screwed something up.

jeroenhd•1h ago

Explicit tail call optimization is in the works but I don't think it's available in stable jut yet.

The `become` keyword has already been reserved and work continues to happen (https://github.com/rust-lang/rust/issues/112788). If you enable #![feature(explicit_tail_calls)] you can already use the feature in the nightly compiler: https://play.rust-lang.org/?version=nightly&mode=debug&editi...

(Note that enabling release mode on that link will have the compiler pre-calculate the result so you need to put it to debug mode if you want to see the assembly this generates)

yodacola•1h ago

FlatBuffers are already faster than that. But that's not why we choose Protobuf. It's because a megacorp maintains it.

nindalf•1h ago

You're saying we choose Protobufs [1] because Google maintains it but not FlatBuffers [2]?

[1] - https://github.com/protocolbuffers/protobuf: Google's data interchange format

[2] - https://github.com/google/flatbuffers: Also maintained by Google

rafaelmn•58m ago

I get the OP is off base with his remark - but at the same time maintained by Google means shit in practice.

AFAIK they have a bunch of production infra on protobuff/gRPC - not so sure about flatbufferrs which came out of the game dev side - that's the difference maker to me - which project is actually rooted in.

dewey•51m ago

> but at the same time maintained by Google means shit in practice.

If you worked on Go projects that import Google protobuf / grpc / Kubernetes client libraries you are often reminded of that fact.

secondcoming•46m ago

Yet they've yet to release their internal optimisation that allows zero-copying string-type fields.

rozenmd•1h ago

"5 times faster" reminds me of Cap'n Proto's claim: in benchmarks, Cap’n Proto is INFINITY TIMES faster than Protocol Buffers: https://capnproto.org/

7777332215•46m ago

In my experience capn proto is much less ergonomic.

gf000•17m ago

I mean, cap'n'proto is written by the same person who created protobuf, so they are legit (and that somewhat jokish claim is simply that it requires no parsing).

t-writescode•52m ago

Just for fun, how often do regular-sized companies that deal in regular-sized traffic need Protobuf to accomplish their goals in the first place, compared to JSON or even XML with basic string marshalling?

tcfhgj•47m ago

Well, protobuf allows to generate easy to use code for parsing defined data and service stubs for many languages and is one of the faster and less bandwidth wasting options

bluGill•32m ago

In most languages protobuf is eaiser because it generates the boilerplate. And protobuf is cross language so even if you are working in javascript where json is native protobuf is still faster because the other side can be whatever and you are not spending their time parsing.

steeve•47m ago

tldr: they replaced using protobuf as the type system across language boundaries for FFI with true FFI

Xunjin•35m ago

I loved, every clickbait title should come with a tldr just like this one.

lowdownbutter•40m ago

Don't read clickbaity headlines and scan hacker news five times faster.

Show HN: Carlton × CMP Signature AR NUME

The Inverse DevOps Principle

Major Canadian computer hardware online store compromised for months

Hyundai Motor's Korean union warns of humanoid robot plan, sees threat to jobs

A Management Philosopher with Heady Ideas About Beer (2009)

Show HN: Botnet of Ares – Hacking Simulator Open Playtest

Show HN: ObsessionDB – We rebuilt ClickHouse infrastructure to cut our costs 50%

Ask HN: What AI feature looked in demos and failed in real usage? Why?

Ask HN: Anti-John the Baptist?

Show HN: Build agents via YAML with Prolog validation and 110 built-in tools

AI is not a NOT a horse (2023)

Partitioning a 17TB Table in PostgreSQL

VS Code: Broken rendering on macOS after app resumed from idle state

OpenAI Wants a Cut of Your Profits: Inside Its New Royalty-Based Plan

Shenzhou-20 Returns Safely After Historic In-Flight Debris Repairs

Alternatives to MinIO for single-node local S3

Show HN: A verified foundation of mathematics in Coq (Theory of Systems)

Heathrow's new scanners end dreaded rummage for liquids and laptops

Can the prescription drug leucovorin treat autism? History says, probably not

Davos Stops Pretending

For the Children: A short story about the endgame of EU Chat Control

An Adversarial Coding Test

Go Developer Survey 2025: How Gophers Use AI Tools, Editors, and Cloud Platforms

Ask HN: What's the current best local/open speech-to-speech setup?

A Multi-Entry Control Flow Graph Design Conundrum

Bernstein vs. United States

Show HN: Workmux – Parallel development in tmux with Git worktrees

Show HN: 9 years building an open-source financial platform

Ask HN: What 'AI feature' created negative ROI in production?

TigerBeetle's Stablecoin Mistake