One isn't better than the other. But LiteFS isn't a "distributed SQLite" in the sense you'd think of with rqlite. It's a system for getting read-only replicas, the same way you've been able to do with log shipping on n-tier databases for decades.
Am I understanding correctly that I will be able to restore a database to any point-in-time that is while the litestream process is running? Because auto-checkpointing could consume the WAL while it isn't running?
So for an extreme example if the process crashed for an hour between 2:00 and 3:00, I could restore to 1:55 or 3:05 but the information required to restore between 2:00 and 3:00 is lost?
I can't wait to try this out.
Recently, I deployed a little side project using a small postgres vm on fly.io After a couple of days, and only having about 500kb of data stored in that db, the postgres vm went into an unrecoverable fail loop, saying it ran out of memory, restarting, then immediately running out of memory again, so on and so forth
It took about 3-4hrs to recover the data jumping through a lot of hoops to be able to access the data, copy it to another volume and finally download it
I would've reached for support, but it seems like the only option available is just posting on their forum. I saw a couple of related posts, all with unsatisfactory answers unfortunately
To be fair, it was incredibly easy to get up and running with them. On the other hand, almost all the time I saved by that quick start, was wasted recovering the failing db, all the while my site was down
Ironically, I originally developed the project using sqlite, but then switched to postgres to deploy
For Postgres I ended up going with Neon (neon.tech), very happy with them so far. Super easy to setup and get up and running, also love being able to just easily see the data from their web interface
People sometimes have a hard time with the idea that we write about things because they are interesting to us, and for no other reason. That's also 60-70% of why Ben does what he does on Litestream.
We have a sneaking suspicion that the robots that write LLM code are going to like SQLite too. We think what coding agents like Phoenix.new want is a way to try out code on live data, screw it up, and then rollback both the code and the state.
Agents would of course work well if they can go back in time to checkpoints and branch from there, exploring solutions parallely as needed.
Anyone who has experience with building workflows (Amazon SWF, Temporal, and the like) knows how difficult it is to maintain determinism in face of retries & re-drives in multi-tier setups (especially, those involving databases).
Replit recently announced their Agent's integration with Neon's time travel feature [0] for exactly the purpose outlined in TFA. Unlike Fly.io though, Replit is built on GCP and other 3p providers like Neon and it is unclear if both GCP & Databricks won't go all Oracle on them.
Can this be done with only Litestream, or is LiteVFS still in development? I looked into this last year but was put off by LiteFS's stated write performance penalty due to FUSE [1]; it's still marked as WIP [2] and hasn't seen updates for over a year.
[1] https://fly.io/docs/litefs/faq/#what-are-the-tradeoffs-of-us...
1. A built-in UI and CLI that manages SQLite from a volume. Getting the initial database on a Fly Machine requires more work than it should.
2. `fly console` doesn't work with SQLite because it spins up a separate machine, which isn't connected to the same volume where the SQLite data resides. Instead you have to know to run `fly ssh console —pty`, which effectively SSH's into the machine with the database.
The problem in general with SQLite web apps is they tend to be small apps, so you need a lot of them to make a decent amount of money hosting them.
In the meantime, it's pretty straightforward to use as a library.
we're using it on production for a write-heavy interal use-case (~12GB compressed) for more than a year now; and it's costing us a couple hundred pennies per month (azure).
excited to try the new changes when they land.
You might start running into problems at tens or hundreds of thousands of writes a second, though even then you may be OK on the right hardware.
maxmcd•5h ago
Really nice to see this, I wrote this comment almost 2 years ago when I was a little miffed about trying to use litestream and litefs: https://news.ycombinator.com/item?id=37614193
I think this solves most of the issues? You can now freely run litestream on your DB and not worry about issues with multiple writers? I wonder how the handoff is handled.
The read replica FUSE layer sounds like a real nice thing to have.
edit: Ah, it works like this: https://github.com/benbjohnson/litestream/pull/617
> When another Litestream process starts up and sees an existing lease, it will continually retry the lease acquisition every second until it succeeds. This low retry interval allows for rolling restarts to come online quickly.
Sounds workable!