Post talks about how to use io_uring, in the context of building a "database" (a demonstration key-value cache with a write-ahead log), to maintain durability.

The recovery process is to "only apply operations that have both intent and completion records." But then I don't see the point of logging the intent record separately. If no completion is logged, the intent is ignored. So you could log the two together.

Presumably the intent record is large (containing the key-value data) while the completion record is tiny (containing just the index of the intent record). Is the point that the completion record write is guaranteed to be atomic because it fits in a disk sector, while the intent record doesn't?

    Write intent record (async)
    Perform operation in memory
    Write completion record (async)
    * * Wait for intent and completion to be flushed to disk * *
    Return success to client

    * * Wait for intent and completion to be flushed to disk * *

“Write intent record (async) Perform operation in memory Write completion record (async) Return success to client

During recovery, I only apply operations that have both intent and completion records. This ensures consistency while allowing much higher throughput. “

Does this mean that a client could receive a success for a request, which if the system crashed immediately afterwards, when replayed, wouldn’t necessarily have that request recorded?

How does that not violate ACID?

Slightly off topic but anyone knows when/if Google is going to enable io_uring for Android?

Great to see someone going into this. I wanted to do a simple LSM tree using io_uring in Zig for some time but couldn't get into it yet.

I always use this approach for crash-resistance:

- Append to the data (WAL) file normally.

- Have a seperate small file that is like a hash + length for WAL state.

- First append to WAL file.

- Start fsync call on the WAL file, create a new hash/length file with different name and fsync it in parallel.

- Rename the length file onto the real one for making sure it is fully atomic.

- Update in-memory state to reflect the files and return from the write function call.

Curious if anyone knows tradeoffs between this and doing double WAL. Maybe doing fsync on everything is too slow to maintain fast writes?

I learned about append/rename approach from this article in case anyone is interested:

- https://discuss.hypermode.com/t/making-badger-crash-resilien...

- https://research.cs.wisc.edu/adsl/Publications/alice-osdi14....

I don't get this. How can two(+) WAL operations be faster than one (double the sync IOPS)?

I think this database doesn't have durability at all.

I don't get this scheme at all. The protocol violates durability, because once the client receives success from server, it should be durable. However, completion record is async, it is possible that it never completes and server crashes.

During recovery, since the server applies only the operations which have both records, you will not recover a record which was successful to the client.

Great article, but I have a question:

The problem with naive async I/O in a database context at least, is that you lose the durability guarantee that makes databases useful. When a client receives a success response, their expectation is the data will survive a system crash. But with async I/O, by the time you send that response, the data might still be sitting in kernel buffers, not yet written to stable storage.

Shouldn't you just tie the successful response to a successful fsync?

Async or sync, I'm not sure what's different here.

First, I think the article provides false claim, the solution doesn't guarantee durability. Second, I believe good synchronous code is better than bad asynchronous code, and it's way easier to write good synchronous code than asynchronous code, especially with io_uring. Modern NVMe are fast, even with synchronous IO, enough for most applications. Before thinking about asynchronous, make sure your application use synchronous IO well.

To be clear, this is different to what we do (and why we do it) in TigerBeetle.

For example, we never externalize commits without full fsync, to preserve durability [0].

Further, the motivation for why TigerBeetle has both a prepare WAL plus a header WAL is different, not performance (we get performance elsewhere, through batching) but correctness, cf. “Protocol-Aware Recovery for Consensus-Based Storage” [1].

Finally, TigerBeetle's recovery is more intricate, we do all this to survive TigerBeetle's storage fault model. You can read the actual code here [2] and Kyle's Jepsen report on TigerBeetle also provides an excellent overview [3].

[0] https://www.youtube.com/watch?v=tRgvaqpQPwE

[1] https://www.usenix.org/system/files/conference/fast18/fast18...

[2] https://github.com/tigerbeetle/tigerbeetle/blob/main/src/vsr...

[3] https://jepsen.io/analyses/tigerbeetle-0.16.11.pdf