This is also a fairly good way to handle large amounts of data with maximum performance on spinning rust, and at the heart of systems like Kafka.
I had assumed that the story would be very different with SSDs, so it's surprising to see append only logs show up again.
And a text file is the most useful general format - easy to write, easy to process with standard tools.
I've used this in the past on data sets of hundreds of millions of lines, maybe billions.
It's also true that you could use a memory-mapped indexed file for faster searches - I've used sqlite for this.
craftkiller•4w ago
Did you use a tool to create them, and if so, what is that tool?
agavra•4w ago
cipehr•4w ago