Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•11mo ago

Comments

nine_k•11mo ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•11mo ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•11mo ago

They mention this in the article.

spockz•11mo ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•11mo ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•11mo ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•11mo ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•11mo ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•11mo ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•11mo ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•11mo ago

This was very well written. Excellent article!

NetOpWibby•11mo ago

Is this like MessagePack for Haskell?

Show HN: A document snapshot solution to counter backups with name "Final_final"

Humanoid Robot Shatters Human Half-Marathon Record to Make History

American Library Association releases list of the most challenged books of 2025

The glance, and song, that could define the Premier League title race

US Class Action Suit over BrowserGate

Show HN: Clock-MCP, stop guessing the time in AI

We got 207 tok/s with Qwen3.5-27B on an RTX 3090

After a 7.7-magnitude quake, Japan warns of slightly higher risk of a mega-quake

Personal AI Safety: The Default Settings Will Not Save You

Pasta sauce wants to record your family

Everything is AI now – does this kill the excitment of software development?

America's coming revolt is in the 'wired belt'

Show HN: Git Push No-Mistakes

Kimi vendor verifier – verify accuracy of inference providers

Expansion Artifacts

A Nation of Copycats in a Data Center

Some fish can tell when you're staring at them or their eggs

Scientists Uncover Potential Brain Risks of Popular Fish Oil Supplements

Freedom of Information Request – Chat GPT Conversations – UK Science Secretary

A Camera That Freezes Rotation | Stroboscopic Video

Prediction Markets: Last Week Tonight with John Oliver [video]

Alignment by Default?

Eight Rules to Regain Public Trust in Academia

Knausgaard's Diabolic Realism

As Enrollment Dips, School Administrators Turn to TikTok to Advertise

Changes to GitHub Copilot Individual Plans

Polymarket profits by placing bets on its own platform [video]

Agentic Coding Is About to Fracture Open Source

Show HN: troubleshooting.sh – paste a error, get a fix with root cause explained

Debasement