Wow, with prior knowledge of a bug our tool found it too!
Great work pal, I’m sure the phone is gonna ring off the hook.
Oh btw, we also found some other ones… Stay tuned!
It states you replicated the bug once the SQLite team fixed it, and published it.
Not sure what’s difficult about replicating behaviour when it’s spelled out for you.
My understanding is that this workload is very uncommon: the Tailscale blog says they used a custom unusual configuration to have many checkpoints like this. So without knowing the bug, it seems unlikely one would build this workload and then find the bug. If that makes sense.
Edit: just want to say that you being able to repro it is awesome, but that the overall claim seems a little overstated to me.
I'm only sad that I didn't put SQLite under test earlier in the year, or I would have found this issue right away. If you look at the workload[1], you can see how simple it is. Exactly the same kind of workload we write every day to help our customers find bugs.
If you have any questions about our process or how debugging with Antithesis works, please let me know! Thanks for reading!
[1]: https://github.com/antithesishq/sqlite/blob/3.51.2-instrumen...
The workload I put in place does exactly that[1]. It runs a write workload from multiple processes concurrently on the same SQLite database to cause writes to build up in the WAL, and runs checkpoints concurrently. This exercises the portion of the WAL code that, from a trivial read-through, is most likely to contain bugs (and turns out, did!).
Said differently, this is exactly the approach we take with all stateful transactional systems. I am only sad that I didn't do this experiment months earlier, as it would have saved Tailscale and the SQLite team a lot of time.
[1]: https://github.com/antithesishq/sqlite/blob/3.51.2-instrumen...
I couldn't find it, so its unclear if the prompt was completely "make a test suite" or was lead towards finding it in the first place, which wouldn't be a fair test.
The closest I mention of the prompt I could find was:
> Then I asked it to write a simple workload which exercised the WAL insert and checkpoint code. Notably, this is a completely generic workload.
With a skeptical lens, unclear.
Looking forward to seeing the other bugs they found.
I'd be interested in seeing what the SQLite people have to say about them if/when they are submitted upstream
this one falls very flat on that front imo. the "antithesis finds rare bugs" is the overwhelming majority of the actual content here, where as the usual meat of the blog, the hard work, the deep thinking, really distills down to "typed this into claude mobile, typed that into claude mobile". yes, its a statement to the power of the technologies of today that's all it takes to find this bug, but man it's a sad signal for those of us looking for compelling technical reads instead of b2b sales pitches.
https://news.ycombinator.com/item?id=49278424 https://news.ycombinator.com/item?id=49278521
Also appreciate the nice words at the end :) I'm feeling a bit ganged up on.
As-is, this is just 20/20 hindsight with concerns about leading the AI on through the prompt hand-waved away. Come on.
Suuure. So why exactly haven't you found this bug already a long time ago? I mean it is trivially obvious that this is where bugs in SQLite would be, right?
SQLite is open source and is one of the most popular databases, surely testing it to find bugs would also be obvious?
My goal, which I believe I accomplished, was to determine if Antithesis would have found this bug if we had put the SQLite WAL under test.
1. The specific bug isnt mentioned 2. (If youre game) a model with a knowledge-cutoff date before the report is used
Which is: "Given the knowledge of a kind of bug that can happen in one Raft implementation, can a similar workload find similar bugs in other Raft implementations?" Turns out it can.
john_strinlai•1h ago