They did not process 178 billion rows per second. They did a search that found something in a large data set by eliminating the parts of the data set that could not have contained the item. Same way I did by picking one grocery store and going straight to the shelf.
So, the analogy doesn't really hold true unless you actually have these trillions of alternate products stored in your brain and manage to cite the matching subset on demand.
If I have 10 billion rows in an SQL database, with a UNIQUE index, and do SELECT * FROM t1 WHERE pk=<number>, then I have “processed” 10 billion rows.
If I do 10k of these queries per second, I have processed 100T rows per second.
It's not just a tech blog post - it's a thriller. ;)
https://en.wikipedia.org/wiki/Bloom_filter
In the 80's or so when I thought I was being really clever I came up with another variation on this and I recall being quite annoyed when someone on HN pointed out (many years later) that this was a staple of computing science for longer than that I had been busy with computers. So much for having original thoughts...
dmitrygr•4mo ago