I ran into memory issues with a high-load project so I built a compact binary encoder/decoder for Pydantic models. It cut in-RAM object size by up to 7× vs json.dumps(), and ended up saving the whole service from collapsing.
Works with annotated Pydantic models and gives you:
– .serialize() -> bytes
– .deserialize(bytes) -> Model
Curious to hear whether others here have hit similar problems and how you solved it?
P.S. Project was a Telegram MiniApp with 10m+ MAU, we were storing cached user objects in Redis Cluster
Locutus_•3mo ago
I've had exactly the same situation, ~2M MAU service with REDIS as the only persistence system, all data being JSON serialized Pydantic models. The storage overhead was just terrible and cost real money.
This would have been a super nice to have back then.
I wonder though how much sense it would make to get something like this mainlined into upstream Pydantic? as having this downstream would give many continuity and dependency lock concerns. And having it as part of the main library would significantly drive adoption rate.
sijokun•3mo ago
Thanks, your Redis story sounds exactly like my motivation. When the cache layer becomes the primary persistence and you’re pushing around millions of JSON-encoded Pydantic objects, the overhead hits both RAM and the bill very quickly. That pain is real.
Regarding getting this into Pydantic core: I doubt the maintainers would want to ship and maintain a binary serialization layer inside the main project. That said, it might still be worth opening an issue to hear their perspective - worst case, it becomes a documented "won’t include".
As for continuity and lock-in concerns - my primary production workload depends on this library, so I fully intend to keep it updated alongside new Pydantic and Python releases. I also intentionally designed it with minimal dependencies: core functionality uses only pydantic and typing-extensions. The only extra is leb128, and that’s just for UInt128, which almost nobody truly needs - I’m considering moving that to an optional extra to keep the base install even leaner.
sijokun•3mo ago
Found in issue where maintainer answered that they are only going to support JSON
sijokun•3mo ago
GitHub: https://github.com/sijokun/PyByntic
Works with annotated Pydantic models and gives you: – .serialize() -> bytes – .deserialize(bytes) -> Model
Curious to hear whether others here have hit similar problems and how you solved it?
P.S. Project was a Telegram MiniApp with 10m+ MAU, we were storing cached user objects in Redis Cluster
Locutus_•3mo ago
This would have been a super nice to have back then.
I wonder though how much sense it would make to get something like this mainlined into upstream Pydantic? as having this downstream would give many continuity and dependency lock concerns. And having it as part of the main library would significantly drive adoption rate.
sijokun•3mo ago
Regarding getting this into Pydantic core: I doubt the maintainers would want to ship and maintain a binary serialization layer inside the main project. That said, it might still be worth opening an issue to hear their perspective - worst case, it becomes a documented "won’t include".
As for continuity and lock-in concerns - my primary production workload depends on this library, so I fully intend to keep it updated alongside new Pydantic and Python releases. I also intentionally designed it with minimal dependencies: core functionality uses only pydantic and typing-extensions. The only extra is leb128, and that’s just for UInt128, which almost nobody truly needs - I’m considering moving that to an optional extra to keep the base install even leaner.
sijokun•3mo ago
https://github.com/pydantic/pydantic/discussions/4097
Locutus_•3mo ago