Some time ago, I started a project that would require a high performance networking system. Naturally, to send data between computers I also needed a serialization format, but a text format like JSON was just too slow.
I started researching binary formats. But to my surprise, every binary format had major downsides. Almost all of them require schema definitions up front written in an IDL, as well as extra tooling and build steps. Schema evolution is also fragile and hard to maintain backwards compatibility.
It feels like being forced into a choice:
1) JSON: convenient but slow
2) Binary formats: fast but painful to use
Why do I need to choose? I don't want to sacrifice my sanity to get acceptable performance.
I kept looking and found a research paper from 2024 about a conceptual schemaless zero-copy format. Lightning struck. This idea seemed extraordinary: you essentially send a serialized B-tree over the network so that the receiver can perform lookups and mutations directly without parsing. The only problem: nobody had made a (public) implementation yet.
I figured sometimes you have to be the first, so for the last 6 months I've been working on this and am proud to say it is now released and open source under MIT license.
The ability to modify serialized data directly is something not typically supported by other formats. I also extended the original format with JSON compatibility, such that it is now possible to convert between Lite³ and JSON. Despite this, it achieves performance on-par with modern zero-copy formats like Flatbuffers and Cap'n Proto. Right now the implementation is in C, but I plan to create bindings for other languages as well.
So hopefully some fellow developers find it interesting!
Official GitHub repository: https://github.com/fastserial/lite3
Official documentation site: https://lite3.io
Mailing list: see project README