Compression: For anything that ends up large it's probably desired. Though consider both algorithm and 'strength' based on the use case carefully. Even a simple algorithm might make things faster when it comes time to transfer or write to permanent storage. A high cost search to squeeze out yet more redundancy is probably worth it if something will be copied and/or decompressed many times, but might not be worth it for that locally compiled kernel you'll boot at most 10 times before replacing it with another.
>This also allows for easy concatenation.
How would it be easier than putting it at the front?
So if you rewrite an index at the head of the file, you may end up having to rewrite everything that comes afterwards, to push it further down in the file, if it overflows any padding offset. Which makes appending an extremely slow operation.
Whereas seeking to end, and then rewinding, is not nearly as costly.
But if you're writing indices, there's a good chance that you do care about performance.
Which is what dbm, bdb, Windows search indexes, IBM datasets, and so many, many other standards will do.
In theory, files should be just unrolled linked lists (or trees) of bytes, but I guess a lot of internal code still assumes full, aligned blocks.
Have you ever wondered why `tar` is the Tape Archive? Tape. Magnetic recording tape. You stream data to it, and rewinding is Hard, so you put the list of files you just dealt with at the very end. This now-obsolete hardware expectation touches us decades later.
Reading the index from the end of the file is also quick; where you read next depends on what you are trying to find in it, which may not be the start.
The point is, not all is black and white. Where to put the index is just another trade off.
For example, if it is an archival/recording oriented use case, then you make it cheap/easy to add data and possibly add some resiliency for when recording process crashes. If you want efficient random access, streaming, storage efficiency, the same dataset can be stored in a different layout without loss of quality—and conversion between them doesn’t have to be extremely optimal, it just should be possible to implement from spec.
Like, say, you record raw video. You want “all of the quality” and you know all in all it’s going to take terabytes, so bringing excess capacity is basically a given when shooting. Therefore, if some camera maker, in its infinite wisdom, creates a proprietary undocumented format to sliiightly improve on file size but “accidentally” makes it unusable in most software without first converting it using their own proprietary tool, you may justifiedly not appreciate it. (Canon Cinema Raw Light HQ—I kid you not, that’s what it’s called—I’m looking at you.)
On this note, what are the best/accepted approaches out there when it comes to documenting/speccing out file formats? Ideally something generalized enough that it can also handle cases where the “file” is in fact a particularly structured directory (a la macOS app bundle).
A team member just created a new tool that uses the tar format (streamable), but then puts the index as the penultimate entry, with the last entry just being a fixed size entry with the offset of the beginning of the index.
In this way normal tar tools just work but it’s possible to retrieve a listing and access a file randomly. It’s also still possible to append to it in the future, modulo futzing with the index a bit.
(The intended purpose is archiving files that were stored as S3 objects back into S3.?
Think about a format that has all those properties and you've used - PDF. PDFs the size of several 100s of MB aren't rare. Now imagine how it works in your world:
* Add a note? Wait for the file to be completely rewritten and burn 100s of MB of your data to sync to iCloud/Drive.
* Fill a form? Same.
* Add an annotation with your Apple Pencil? Yup, same.
Now look at how it works right now:
- Add a text? Fill a form? Add a drawing? A few KB of data is appended and uploaded.
* Sign the document to confirm authenticy? You got it, a few KB of data at the end.
* Determine which data was added after the document was signed and sign it with another cert? A few bytes.
Do you need to stream the PDF? Load the last chunk to detect the dictionary. If you don't want to do that, configure PDF writer to output the dictionary at the start and you still end up with a better solution.
[1] SQLite's own sqlar format is a bad idea for this reason.
SEE is a proprietary extension, however generous its license is. So it is not very meaningful when sqlar is compared against ZIP. Not to say that I necessarily see encryption as a fundamental feature for compressed archive formats though---I'm advocating for age [1] integration instead.
[1] https://github.com/FiloSottile/age
> IMHO sqlar is not competing with ZIP (can zip do metadata and transactions?)
In my understanding SQLite's support for sqlar and ZIP occurred at the same time, so I believe that sqlar was created to demonstrate an alternative to ZIP (and that the demonstration wasn't good enough). I'm aware that this is just a circumstantial evidence, so let me know if you have some concrete one.
ZIP can of course do metadata in the form of per-file and archive comments. For more structured metadata, you can make use of extra fields if you really, really want, but at that point SQLite would indeed be a better choice. I however doubt it's a typical use case.
ZIP can be partially updated in place but can't do any transaction. But it should be noted that SQLite handles transaction by additional files (`-journal` or `-wal` files). So both sqlar and ZIP would write to an additional file during the update process, though SQLite would write much less data compared to ZIP. Any remaining differences are invisible to end users, unless the in-place update is common enough in which case the use of SQLite is justified.
> In my understanding SQLite's support for sqlar and ZIP occurred at the same time
I believe so too.
I agree with you on SQLAR being poor general-purpose archive or compression format compared to ZIP; what I'm arguing is that its very good file format for certain applications, offering structured, modifiable, and searchable file storage. We had great success using it as db/file format for PLM solution packed both as desktop and web app. Same database can then be used to power the web ui (single tenant SaaS deployments), and for desktop app (web export is simply a working file for desktop app). This file being just a simple sqlite db lets users play with data, do their own imports, migrations etc., while having all files & docs in one place.
It ends up having some overhead compared to established ones, but the ability to query over the attributes of 10000s of files is pretty nice, and definitely faster than the worst case of tar.
My archiver could even keep up with 7z in some cases (for size and access speed).
Implementing it is also not particularly tricky, and SQLite even allows streaming the blobs.
Making readers for such a format seems more accessible to me.
> My archiver could even keep up with 7z in some cases (for size and access speed).
7z might feel slow because it enables solid compression by default, which trades decompression speed with compression ratio. I can't imagine 7z having a similar compression ratio with correct options though, was your input incompressible?
You're right that their size is limited, though, and it's actually worse than you even thought (1 GB).
** The content being read or written might appear on the main page
** or be scattered out on multiple overflow pages.
For my case it happened to work out because it was a CDC based deduplicating format that compressed batches of chunks. Lots of flexibility with working within the limits given that.
The primary goal here was also making the reader as simple as possible whilst still having decent performance.
I think my workload is very unfair towards (typical) compressing archivers: small incremental additions, needs random access, indeed frequent incompressible files, at least if seen in isolation.
I've really brought up 7z because it is good at what it does, it is just (ironically) too flexible for what was needed. There probably some way of getting it to perform way better here.
zpack is probably a better comparison in terms of functionality, but I didn't want to assume familiarity with that one. (Also I can't really keep up with it, my solution is not tweaked to that level, even ignoring the SQLite overhead)
*: A few bytes less, actually; the 1 GB limit is on the total size of a row, including its ID and any other columns you've included.
It just may not always be the most performant option. For example, for map tiles there is alternatively the pmtiles binary format which is optimized for http range requests.
https://shapeof.com/archives/2025/4/acorn_file_format.html
The author notes that an advantage is that other programs can easily read the file format and extract information from it.
> Acorn’s native file format is used to losslessly store layer data, editable text, layer filters, an optional composite of the image, and various metadata. Its advantage over other common formats such as PNG or JPEG is that it preserves all this native information without flattening the layer data or vector graphics.
As I've mentioned, this is a good use case for SQLite as a container. But ZIP would work equally well here.
[1] https://flyingmeat.com/acorn/docs/technotes/ACTN002.html
Your reply suggests that, if all the metadata is auxiliary it can be segregated from the data and doesn't count as a composite.
However, that doesn't exclude archives (in many use-cases the file metadata is as important as the data itself; consider e.g. hardlinks in TAR files)
Nor does it exclude certain vital metadata for images: resolution, color-space, and bit-depth come to mind.
You've got a good point that there are multiple types of metadata and some metadata might be crucial for interpreting data. I would say such "structural" metadata should be considered as a part of data. I'm not saying it is not a metadata; it is a metadata inside some data, so doesn't count for our purpose of defining a composite.
I also don't think tar hardlinks are metadata for our purpose, because it technically consists of the linked path instead of the file contents and the information that the file is a hardlink, where the former is clearly a data and the latter is a metadata used to reconstruct the original file system so should be considered as a part of larger data (in this case, a logical notion of "file").
I believe these examples should be enough to derive my own definition of "composite". Please let me know otherwise.
That wouldn’t support partial parsing.
From the point of view of the article, a SQLite file is similar to a chunked file format: the compact directory of what tables etc. it contains is more heavyweight than listing chunk names and lengths/offsets, but at least as fast, and loading only needed portions of the file is automatically managed.
A friend wanted a newer save viewer/editor for Dragonball Xenoverse 2, because there's about a total of two, and they're slow to update.
I thought it'd be fairly easy to spin up something to read it, because I've spun up a bunch of save editors before, and they're usually trivial.
XV2 save files change over versions. They're also just arrays of structs [0], that don't properly identify themselves, so some parts of them you're just guessing. Each chunk can also contain chunks - some of which are actually a network request to get more chunks from elsewhere in the codebase!
[0] Also encrypted before dumping to disk, but the keys have been known since about the second release, and they've never switched them.
- Agreed that human-readable formats have to be dead simple, otherwise binary formats should be used. Note that textual numbers are surprisingly complex to handle, so any formats with significant number uses should just use binary.
- Chunking is generally good for structuring and incremental parsing, but do not expect it to provide reorderability or back/forward compatibility somehow. Unless explicitly designed, they do not exist. Consider PNG for example; PNG chunks were designed to be quite robust, but nowadays some exceptions [1] do exist. Versioning is much more crucial for that.
[1] https://www.w3.org/TR/png/#animation-information
- Making a new file format from scratch is always difficult. Already mentioned, but you should really consider using existing file formats as a container first. Some formats are even explicitly designed for this purpose, like sBOX [2] or RFC 9277 CBOR-labeled data tags [3].
Especially true of floats!
With binary formats, it's usually enough to only support machines whose floating point representation conforms to IEEE 754, which means you can just memcpy a float variable to or from the file (maybe with some endianness conversion). But writing a floating point parser and serializer which correctly round-trips all floats and where the parser guarantees that it parses to the nearest possible float... That's incredibly tricky.
What I've sometimes done when I'm writing a parser for textual floats is, I parse the input into separate parts (so the integer part, the floating point part, the exponent part), then serialize those parts into some other format which I already have a parser for. So I may serialize them into a JSON-style number and use a JSON library to parse it if I have that handy, or if I don't, I serialize it into a format that's guaranteed to work with strtod regardless of locale. (The C standard does, surprisingly, quite significantly constrain how locales can affect strtod's number parsing.)
if (nativeEndianness != littleEndian) swapBytes(&theFloat);
[^1]: https://cppreference.com/w/c/language/floating_constant.html
For example, `0x1.2p3` represents `9.0`.
Maybe it's discipline-specific, but with the reasonable care in handling floats that most people are taught, I've never had a consequential mishap.
For example, the ASCII string "Morn", stored as the bytes '0b01001101 0b01101111 0b01110010 0b01101110', could be interpreted as the 32-bit float 0b01101110011100100110111101001101, representing the number 1.8757481691240478e+28.
So you couldn't really just have smart "float recognition" built in to an editor as a general feature, you would need some special format which the editor understands which communicates "the following 4 bytes is a single-precision float" or "the following 8-byte is a double-precision float".
E.g. if you are interested in storing significant amounts of structured floating point data, choosing something like HDF5 will not only make your life easier it will also make it easy to communicate what you have done to others.
Is there a reason not to use a lot more characters? If your application's name is MustacheMingle, call the file foo.mustachemingle instead of foo.mumi?
This will decrease the probability of collision to almost zero. I am unaware of any operating systems that don't allow it, and it will be 100% clear to the user which application the file belongs to.
It will be less aesthetically pleasing than a shorter extension, but that's probably mainly a matter of habit. We're just not used to longer file name extensions.
Any reason why this is a bad idea?
The most popular operating system hides it from the user, so clarity would not improve in that case. At leat one other (Linux) doesn't really use "extensions" and instead relies on magic headers inside the files to determine the format.
Otherwise I think the decision is largely aestethic. If you value absolute clarity, then I don't see any reason it won't work, it'll just be a little "ugly"
mostly for executable files.
I doubt many Linux apps look inside a .py file to see if it's actually a JPEG they should build a thumbnail for.
https://wiki.archlinux.org/title/XDG_MIME_Applications
A lot of apps implement this (including most file managers)
It's also one of my favorite oddities on Linux. If you're a Windows user the idea of a database of signatures for filetypes that exists outside the application that "owns" a file type is novel and weird.
If you mean Windows, that's not entirely correct. It defaults to hiding only "known" file extensions, like txt, jpg and such. (Which IMO is even worse than hiding all of them; that would at least be consistent.)
EDIT: Actually, I just checked and apparently an extension, even an exotic one, becomes "known" when it's associated with a program, so your point still stands.
When under pixel pressure, a graphical file manager might choose to prioritize displaying the file extension and truncate only the base filename. This would help the user identify file formats. However, the longer the extension, the less space remains for the base name. So a low-entropy file extension with too many characters can contribute to poor UX.
You could go the whole java way then foo.com.apache.mustachemingle
> Any reason why this is a bad idea
the focus should be on the name, not on the extension.
It’s prone to get cut off in UIs with dedicated columns for file extensions.
As you say, it’s unconventional and therefore risks not being immediately recognized as a file extension.
On the other hand, Java uses .properties as a file extension, so there is some precedent.
Generic, standardized formats like "jpg" and "pdf", and
Application-specific formats like extension files or state files for your program, that you do not wish to share with competitors.
So, double-clicking the file opened it in the application it was made in, but the Mac would also know which other applications could open that file.
Some cop-out (not necessarily in a bad way) file formats:
1. Don’t have a file format, just specify a directory layout instead. Example: CinemaDNG. Throw a bunch of particularly named DNGs (a file for each frame of the footage) in a directory, maybe add some metadata file or a marker, and you’re good. Compared to the likes of CRAW or BRAW, you lose in compression, but gain in interop.
2. Just dump runtime data. Example: Mnemosyne’s old format. Do you use Python? Just dump your state as a Python pickle. (Con: dependency on a particular runtime, good luck rewriting it in Rust.)
3. Almost dump runtime data. Example: Anki, newer Mnemosyne with their SQLite dumps. (Something suggests to me that they might be using SQLite at runtime.) A step up from a pickle in terms of interop, somewhat opens yourself (but also others) to alternative implementations, at least in any runtime that has the means to read SQLite. I hope if you use this you don’t think that the presence of SQL schema makes the format self-documenting.
4. One or more of the above, except also zip or tar it up. Example: VCV, Anki.
When editing a file locally I would prefer to just have it split up in a directory 99% of the time, only exporting to a ZIP to publish it.
Of course it is trivial to write wrapper scripts to keep zipping and unzipping files, and I have done that, but it does feel a bit hacky and should be an unnecessary extra step.
One question I was hoping to ask anyone who thought about these matters: what accepted approaches do exist out there when it comes to documenting/speccing out file formats? Ideally, including the cases where the “file” is in fact a directory with a specific layout.
Be particularly careful with this one as it can potentially vastly expand the attack surface of your program. Not that you shouldn't ever do it, just make sure the deserializer doesn't accept objects/values outside of your spec.
It should be noted (the article does not) that parsing and deserialisation is generally a known weak area and a common source of CVEs, even when pickling is not used. Being more disciplined about it helps, of course.
“Show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won’t usually need your flowcharts; they’ll be obvious.”
— Fred Brooks
I dislike JSON and some other modern formats (even binary formats); they often are just not as good in my opinion. One problem is they tend to insist on using Unicode, and/or on other things (e.g. 32-bit integers where you might need 64-bits). When using a text-based format where binary would do better, it can also be inefficient especially if binary data is included within the text as well, especially if the format does not indicate that it is meant to represent binary data.
However, even if you use an existing format, you should avoid using the existing format badly; using existing formats badly seems to be common. There is also the issue of if the existing format is actually good or not; many formats are not good, for various reasons (some of which I mentioned above, but there are others, depending on the application).
About target hardware, not all software is intended for a specific target hardware, although some is.
For compression, another consideration is: there are general compression schemes as well as being able to make up a compression scheme that is specific for the kind of data that is being compressed.
They also mention file names. However, this can also depend on the target system; e.g. for DOS files you will need to be limited to three characters after the dot. Also, some programs would not need to care about file names in some or all cases (many programs I write don't care about file names).
In theory you could use ASN.1 DER files the same way you would JSON for human-readable formats. In practice, you're better off picking a different format.
Modern evolutions of ASN.1 like ProtoBuf or Cap'n Proto designed for transmitting data across the network might fit this purpose pretty well, too.
On the other hand, using ASN.1 may be a good way to make people trying to reverse engineer your format give up in despair, especially if you start using the quirks ASN.1 DER comes with and change the identifiers.
I wrote a library to read/write DER, which I have found suitable for my uses. (Although, I might change or add some things later, and possibly also some things might be removed too if I think they are unnecessary or cause problems.)
https://github.com/zzo38/scorpion/blob/trunk/asn1/asn1.c https://github.com/zzo38/scorpion/blob/trunk/asn1/asn1.h
(You can complain about it if there is something that you don't like.)
> In theory you could use ASN.1 DER files the same way you would JSON for human-readable formats. In practice, you're better off picking a different format.
I do use ASN.1 DER for some things, because, in my opinion it is (generally) better than JSON, XML, etc.
> Modern evolutions of ASN.1 like ProtoBuf or Cap'n Proto designed for transmitting data across the network might fit this purpose pretty well, too.
I have found them to be unsuitable, with many problems, and that ASN.1 does them better in my experience.
> On the other hand, using ASN.1 may be a good way to make people trying to reverse engineer your format give up in despair, especially if you start using the quirks ASN.1 DER comes with and change the identifiers.
I am not so sure of this.
For Open-Source projects, human readable file formats are actively harmful.
This mostly is motivated by my experience with KiCad. Principally, there are multiple things that the UI does not expose at all (slots in PCB footprint files) where the only way to add them is to manually edit the footprint file in a text editor.
There are some other similar annoyances in the same vein.
Basically, human readable (and therefore editable) file formats wind up being a way for some things to never be exposed thru the UI. This actively leads to the software being less capable.
(The TEMPLATE.DER lump (which is a binary file format and not plain text) in Super ZZ Zero is not exposed anywhere in the UI; you must use an external program to create this lump if you want it. Fortunately that lump is not actually mandatory, and only affects the automatic initial modifications of a new world file based on an existing template.)
However, I think that human readable file formats are harmful for other reasons.
That’s assuming that parsers will honor this, and not just use the fixed offset that worked for the past ten hears. This has happened often enough in the past.
At that point, you're asking for a filesystem inside of a file. And you can literally do exactly that with a filesystem library (FAT32, etc).
The "Chunk your binaries" point is spot on. Creating a huge binary blob that contains everything makes it hard to work with in constrained environments.
Also, +1 for "Document your format". More like "Document everything". Future you will thank you for it for sure.
If your data format contains multiple streams inside, consider ZIP for the container. Enables standard tools, and libraries available in all languages. The compression support is built-in but optional, can be enabled selectively for different entries.
The approach is widely used in practice. MS office files, Java binaries, iOS app store binaries, Android binaries, epub books, chm documentation are all using ZIP container format.
adelpozo•1mo ago
flowerthoughts•1mo ago
Similar is the discussion of delimited fields vs. length prefix. Delimited fields are nicer to write, but length prefixed fields are nicer to read. I think most new formats use length prefixes, so I'd start there. I wrote a blog post about combining the value and length into a VLI that also handles floating point and bit/byte strings: https://tommie.github.io/a/2024/06/small-encoding
lifthrasiir•1mo ago
flowerthoughts•1mo ago