I have always wanted to combine protobufs with the MQTT protocol, with a native de/serialisation logic built in to clients and servers to reduce the amount of message bytes (compared to the JSONs or other text encoding messages).
Lately, I found the time to implement it and decided to go with Zig to learn the language since I hear a lot about it, and said why not implement most of the parts as much as possible (like MQTT stack, epoll/kqueue-based TCP layer, protobuf encoder/decoder, etc.). Of course, with a lot of AI coding.
So, here you are, the ProtoMQ project. A type-safe, bandwidth-efficient protocol based on MQTT and Protobuf to reduce the bloated messages for your edge-devices. It includes a topic-2-scheme map at build time, where you can register your schemes.
To make sure that it is as performant as it sounds, I've created a benchmark suite and tested the server (for now, on loopback) with a variety of scenarios. You can find the benchmark scripts and results for M2 Pro and Raspberry Pi 5 hardware. Here's a small list for a quick look:
- Message throughput with 10-byte messages:
--- 208k msg/s on M2 Pro
--- 147k msg/s on RPi5
- Message throughput with 64 kb messages:
--- 39k msg/s on M2 Pro
--- 27k msg/s on RPi5
- 10,000 concurrent clients' message fan-out:
--- 0.12 sec on M2 Pro
--- 0.21 sec on RPi5
- Sustained load (10 min) throughput:
--- 8,981 msg/s on M2 Pro
--- 9,012 msg/s on RPi5
- Sustained load (10 min) memory growth:
--- 0.16 MB on M2 Pro
--- 0.09 MB on RPi5
It'd be great to hear some feedback from you folks. I might look into the automatic service discovery for supported schemes, runtime-load of the topic-2-scheme or better performance for several metrics. What do you think? What can be done better for the future of the project?
gyokhan•1w ago
I have always wanted to combine protobufs with the MQTT protocol, with a native de/serialisation logic built in to clients and servers to reduce the amount of message bytes (compared to the JSONs or other text encoding messages).
Lately, I found the time to implement it and decided to go with Zig to learn the language since I hear a lot about it, and said why not implement most of the parts as much as possible (like MQTT stack, epoll/kqueue-based TCP layer, protobuf encoder/decoder, etc.). Of course, with a lot of AI coding.
So, here you are, the ProtoMQ project. A type-safe, bandwidth-efficient protocol based on MQTT and Protobuf to reduce the bloated messages for your edge-devices. It includes a topic-2-scheme map at build time, where you can register your schemes.
To make sure that it is as performant as it sounds, I've created a benchmark suite and tested the server (for now, on loopback) with a variety of scenarios. You can find the benchmark scripts and results for M2 Pro and Raspberry Pi 5 hardware. Here's a small list for a quick look:
- Message throughput with 10-byte messages: --- 208k msg/s on M2 Pro --- 147k msg/s on RPi5
- Message throughput with 64 kb messages: --- 39k msg/s on M2 Pro --- 27k msg/s on RPi5
- 10,000 concurrent clients' message fan-out: --- 0.12 sec on M2 Pro --- 0.21 sec on RPi5
- Sustained load (10 min) throughput: --- 8,981 msg/s on M2 Pro --- 9,012 msg/s on RPi5
- Sustained load (10 min) memory growth: --- 0.16 MB on M2 Pro --- 0.09 MB on RPi5
(Source: https://github.com/electricalgorithm/protomq/tree/main/bench...)
It'd be great to hear some feedback from you folks. I might look into the automatic service discovery for supported schemes, runtime-load of the topic-2-scheme or better performance for several metrics. What do you think? What can be done better for the future of the project?
Here's the repository: https://github.com/electricalgorithm/protomq
Thanks already!
Regards, Gyokhan