Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Update: The extension has been approved and is now live on the Chrome Web Store

Bloom Chose the Team Before the Problem and the Research Before the Pitch

Forget the Robot Tax. Tax the Tokens

Wave overhangs let you print without supports [video]

The Qwen MLX Challenge

Boring bits prepare you for the problems that can't be solved by copy pasting

A Swarm of Blood Robots

Show HN: Ask questions about the Swiss train timetable

Day, in One Frame

Haskell: Origins, Evolution, and Future [video]

Knowledge Representation (2013)

AstroUpdate – Real-time space news, 3D ISS tracking and AI briefs

Carrier Pidge – iOS messaging that travels at 100mph

Show HN: dunx – Nest-style DI for Bun without reflect-metadata

A receipt printer exposed to the internet, inviting people to send it things

Climbing Guide as a Shared Infrastructure

Android Strings XML Translation Online: The Tool That Preserves Your Variables

Show HN: HN Glance – a visual reader for Hacker News links and discussions

We cut RAG costs 5x without losing quality

Why Chinese Citizens Are More Optimistic About AI Than Americans

A Generative Cramér-Rao Bound

AI Paper Trail: What does AI know about you?

How to Invest in AI (2025)

Twitch Adds Opt-Out for Amazon AI Training After Enrolling All Creators

Anthropic shares more details about how Claude's new watermarks will work

Secrets live in six places and one of them is a DM

A Taxonomy of Moats (2019)

A diagnostic for the "be more proactive" performance review

Show HN: Sol-Luna – adaptive Codex orchestration that can choose zero workers

The Experience Feedback Loop