frontpage.
newsnewestaskshowjobs

Made with ♥ by @iamnishanth

Open Source @Github

fp.

Open in hackernews

Making Postgres Distributed with FoundationDB

https://fabianlindfors.se/blog/making-postgres-distributed/
45•emptysea•8mo ago

Comments

fabianlindfors•8mo ago
This is my project, thanks a lot for posting! If anybody happens to have questions, I'm happy to answer them
dev-ns8•8mo ago
Super interesting project! I have zero experience writing Postgres extensions, but I was browsing a blog post about an extension the other day. I was curious what ones workflow looked like while developing a PG extension. From the little I read, it looked like you would have to recompile postgres, including your extension, then enable it just to run it. Seems like a very long feedback loop. Was I missing something?
fabianlindfors•8mo ago
Glad you like it!

You don't need to recompile it actually! It's enough to compile your extension and Postgres can then dynamically link to it, which makes for a perfectly good feedback loop. pgfdb is built with pgrx: https://github.com/pgcentralfoundation/pgrx, which is a framework for building Postgres extensions in Rust that handles all the compiling and linking for you. Highly recommend that if you want to try writing extensions and are also a fan of Rust!

amw-zero•8mo ago
How do table access methods work with standard PG concepts such as the shared_buffers cache? Is that irrelevant since data is stored in FDB? Also, how do the deployment semantics of FDB affect this. If I remember correctly, you typically run one FDB process per CPU on a machine. How do PG processes map to those?
fabianlindfors•8mo ago
A table access method can decide itself if it wants to use the PG buffer cache, pgfdb does not so it's irrelevant in that case.

> Also, how do the deployment semantics of FDB affect this. If I remember correctly, you typically run one FDB process per CPU on a machine. How do PG processes map to those

You would run the two separately, so PG processes would run and scale separately from FDB processes. Postgres spawns a new process for each connection so you would not want to map Postgres itself 1:1 with CPUs!

bithavoc•8mo ago
This is interesting, it would be like YugabyteDB but on top of FoundationDB.
atombender•8mo ago
This is a fun project.

From what I can tell, this doesn't replace the lower level page heap storage, but instead actually provides new implementation of table and indexes through a plugin mechanism called table access methods and index access methods, respectively. This looks very similar to the virtual table mechanism in SQLite, for example, but the C API looks much nicer.

I've not paid much attention to the Postgres extension API, but I'm pleasantly surprised it's that flexible. I've been hearing for years about how Postgres pluggable engine interface isn't flexible enough to implement certain features, but it actually looks really rich. Maybe some of those improvements come from recent work by the OrioleDB people, and others like Citus who develop various alternative table engines?

amw-zero•8mo ago
> From what I can tell, this doesn't replace the lower level page heap storage, but instead actually provides new implementation of table and indexes

These seem contradictory. If the data is stored in FoundationDB, then it won't be stored in the filesystem as blocks, right?

atombender•8mo ago
Correct, my point is that it replaces storage at a higher level.

For example, mvsqlite implements a SQLite VFS that maps page to FoundationDB keys. This means that once the VFS is in place, everything is in FDB. But this means that you have to deal with page conflicts if you want multiple writers to mount the same database, so it's a somewhat coarse-grained system.

What this project is doing allows one to represent each row (tuple) as a FDB key/value, giving you finer-grained control. But it comes at a cost of having to implement all the access methods, including index scans, in terms of FDB.

This is presumably why database metadata (DDL) isn't currently persisted, because those structures aren't normal tables.

fabianlindfors•8mo ago
> What this project is doing allows one to represent each row (tuple) as a FDB key/value, giving you finer-grained control. But it comes at a cost of having to implement all the access methods, including index scans, in terms of FDB.

That's absolutely right! mvsqlite is a cool project but it doesn't make good use of FoundationDB in a way, which I find a bit unfortunate what a nice piece of software it is!

> This is presumably why database metadata (DDL) isn't currently persisted, because those structures aren't normal tables.

Yes, although perhaps a fun fact that all database metadata in Postgres is actually stored as standard tables, the so called system catalog: https://www.postgresql.org/docs/current/catalogs.html.

I'm still mulling over how to implement DDL persistence but one possible way would be to change the actual system catalog tables to be backed by FDB instead, and rely on some cache on the nodes to avoid round tripping to FDB to get metadata for each query.

As you can imagine though the system catalog is quite deeply intertwined with Postgres as a whole so remains to be seen if this is even doable. The alternative would be a more complicated design where the data is stored in some custom format in FDB and then synced by each node into the system catalog.

atombender•8mo ago
That sounds complicated. Presumably you could use watches to quickly get notified on catalog changes, but it suggests that any node could potentially have outdated metadata leading to weird and subtle consistency issues.

With some things like statistics, maybe not a big problem, but with things like dropping columns from tables, you could end up in a situation where a query sees the old table definition.

fabianlindfors•8mo ago
Certainly complicated! This is one of those places where I don't think Postgres compatibility is achievable or even desirable, schema changes in a distributed databases are a whole different beast (and very complicated in large Postgres deployments as well).

It will probably end up being something similar to online schema changes in Cockroach where the changes that can be made are limited for safety and run as a background job that can be tracked: https://www.cockroachlabs.com/docs/stable/online-schema-chan...

I've got a bit of experience with schema changes from building Reshape: https://github.com/fabianlindfors/reshape. I'm hoping to transfer over some concepts from there into pgfdb eventually!

amw-zero•8mo ago
> this doesn't replace the lower level page heap storage

So this is wrong. The heap storage is replaced.

atombender•8mo ago
You miss my point. It's not replacing Postgres' entire block storage with FoundationDB, it's injecting Table Access Methods and Index Access Methods handlers to override tuple storage at the table and index level. These handlers deal with tuples, not blocks.

The author replied to this thread and confirmed this.

fabianlindfors•8mo ago
> I've not paid much attention to the Postgres extension API, but I'm pleasantly surprised it's that flexible. I've been hearing for years about how Postgres pluggable engine interface isn't flexible enough to implement certain features, but it actually looks really rich. Maybe some of those improvements come from recent work by the OrioleDB people, and others like Citus who develop various alternative table engines?

I was also pleasantly surprised by this! I've actually been working on this project for more than a year now with a few false starts trying to find the "right" way to integrate with Postgres that fits well enough with FoundationDB.

Yes! There is a lot of ongoing work on Postgres extensibility and it keeps getting better. The ecosystem is really amazing. I'm for example excited about work being done by Enterprise DB to make custom index access method more generically pluggable, which would allow them to be used for primary keys amongst other things: https://www.postgresql.org/message-id/flat/E72EAA49-354D-4C2...

atombender•8mo ago
I suppose that means you can't implement primary keys until the EnterpriseDB work gets merged?
fabianlindfors•8mo ago
That's right. Largely the same result can be achieved with a unique index though, which I'm working on adding support for right now.

Show HN: Solving NP-Complete Structures via Information Noise Subtraction (P=NP)

https://zenodo.org/records/18395618
1•alemonti06•2m ago•0 comments

Cook New Emojis

https://emoji.supply/kitchen/
1•vasanthv•4m ago•0 comments

Show HN: LoKey Typer – A calm typing practice app with ambient soundscapes

https://mcp-tool-shop-org.github.io/LoKey-Typer/
1•mikeyfrilot•7m ago•0 comments

Long-Sought Proof Tames Some of Math's Unruliest Equations

https://www.quantamagazine.org/long-sought-proof-tames-some-of-maths-unruliest-equations-20260206/
1•asplake•8m ago•0 comments

Hacking the last Z80 computer – FOSDEM 2026 [video]

https://fosdem.org/2026/schedule/event/FEHLHY-hacking_the_last_z80_computer_ever_made/
1•michalpleban•8m ago•0 comments

Browser-use for Node.js v0.2.0: TS AI browser automation parity with PY v0.5.11

https://github.com/webllm/browser-use
1•unadlib•9m ago•0 comments

Michael Pollan Says Humanity Is About to Undergo a Revolutionary Change

https://www.nytimes.com/2026/02/07/magazine/michael-pollan-interview.html
1•mitchbob•10m ago•1 comments

Software Engineering Is Back

https://blog.alaindichiappari.dev/p/software-engineering-is-back
1•alainrk•11m ago•0 comments

Storyship: Turn Screen Recordings into Professional Demos

https://storyship.app/
1•JohnsonZou6523•11m ago•0 comments

Reputation Scores for GitHub Accounts

https://shkspr.mobi/blog/2026/02/reputation-scores-for-github-accounts/
1•edent•14m ago•0 comments

A BSOD for All Seasons – Send Bad News via a Kernel Panic

https://bsod-fas.pages.dev/
1•keepamovin•18m ago•0 comments

Show HN: I got tired of copy-pasting between Claude windows, so I built Orcha

https://orcha.nl
1•buildingwdavid•18m ago•0 comments

Omarchy First Impressions

https://brianlovin.com/writing/omarchy-first-impressions-CEEstJk
2•tosh•23m ago•1 comments

Reinforcement Learning from Human Feedback

https://arxiv.org/abs/2504.12501
2•onurkanbkrc•24m ago•0 comments

Show HN: Versor – The "Unbending" Paradigm for Geometric Deep Learning

https://github.com/Concode0/Versor
1•concode0•25m ago•1 comments

Show HN: HypothesisHub – An open API where AI agents collaborate on medical res

https://medresearch-ai.org/hypotheses-hub/
1•panossk•28m ago•0 comments

Big Tech vs. OpenClaw

https://www.jakequist.com/thoughts/big-tech-vs-openclaw/
1•headalgorithm•30m ago•0 comments

Anofox Forecast

https://anofox.com/docs/forecast/
1•marklit•30m ago•0 comments

Ask HN: How do you figure out where data lives across 100 microservices?

1•doodledood•31m ago•0 comments

Motus: A Unified Latent Action World Model

https://arxiv.org/abs/2512.13030
1•mnming•31m ago•0 comments

Rotten Tomatoes Desperately Claims 'Impossible' Rating for 'Melania' Is Real

https://www.thedailybeast.com/obsessed/rotten-tomatoes-desperately-claims-impossible-rating-for-m...
3•juujian•33m ago•2 comments

The protein denitrosylase SCoR2 regulates lipogenesis and fat storage [pdf]

https://www.science.org/doi/10.1126/scisignal.adv0660
1•thunderbong•34m ago•0 comments

Los Alamos Primer

https://blog.szczepan.org/blog/los-alamos-primer/
1•alkyon•37m ago•0 comments

NewASM Virtual Machine

https://github.com/bracesoftware/newasm
2•DEntisT_•39m ago•0 comments

Terminal-Bench 2.0 Leaderboard

https://www.tbench.ai/leaderboard/terminal-bench/2.0
2•tosh•39m ago•0 comments

I vibe coded a BBS bank with a real working ledger

https://mini-ledger.exe.xyz/
1•simonvc•39m ago•1 comments

The Path to Mojo 1.0

https://www.modular.com/blog/the-path-to-mojo-1-0
1•tosh•42m ago•0 comments

Show HN: I'm 75, building an OSS Virtual Protest Protocol for digital activism

https://github.com/voice-of-japan/Virtual-Protest-Protocol/blob/main/README.md
5•sakanakana00•46m ago•1 comments

Show HN: I built Divvy to split restaurant bills from a photo

https://divvyai.app/
3•pieterdy•48m ago•0 comments

Hot Reloading in Rust? Subsecond and Dioxus to the Rescue

https://codethoughts.io/posts/2026-02-07-rust-hot-reloading/
4•Tehnix•48m ago•1 comments