Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Commission on Federal Paperwork: Drawings by Carol

Denmark abandons plans to ditch Microsoft Office after technical problems

Incompleteness and Computability – An Open Introduction to Gödel's Theorems

Conway's Game of Life

Why popular airplanes are getting scrapped years early

Ask HN: What are your most ambitious projects in 2026?

Show HN: Music player for visually exploring music collection

Making DNS_Erlang Fast: Binary Tricks for the Beam

America's Most Illegal Military Base [video]

J YouTuber and K Programmer Michal Wallace on Narrative Programming

The BBC Strikes Back

eMudhra faces a bigger test of public trust after security lapse

80% to 90% of women reproduce, but only about 40% of men do

Modd: Flexible developer tool that runs processes,responds to filesystem changes

Show HN: Free Local Letterboxd Analytics

DHH's keynote at Rails World: "The most confusing funeral I've ever experienced"

Intel 1103: The Tiny Chip That Killed Magnetic Core Memory and Saved Intel

Musk claims Ukraine violated Starlink service agreement

Black October

Show HN: Independent Vet – Find locally owned vets (and corporate ones) near you

Weight-loss drug reduces sleep apnea in two late-stage trials

The Trouble with the View from Above (2010)

Show HN: Git-ing Over it, learn Git like you'll use it IRL

Just Dashboard open-source dashboard for managing your VPS

Manage Your Dotfiles with Nothing but Git

Retroviral insertions contributed to divergence of human and chimpanzee brains

Show HN: Dadgpt Open Source Halloween Costume

iPhone apps no better for privacy than Android, Oxford study finds (2021)

Backd / Discovery

Five Qwen3.8-27B models match Fable 5 Using GVS5H Harness