Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Ask HN: When you code with AI, do you tell it how to manage state?

AI video in courtroom sparks major legal battle

Ford CEO Says Europe Is Lost to Chinese Electric Vehicles

Show HN: Sumi – ink landscapes that move, with sound synthesised on the device

AI Bots Appeared After Reddit Partnered with OpenAI

Why you do so much and get so little. Read this if you feel stuck

Scientific Overview of Asteroid Analysis

BigHappyFriend Likes Humans (2025)

Show HN: ShipSlides – turn a URL into editable slides with a serverless agent

Show HN: AgentMachinist: make your coding agent show its work

Pretrained Deep Learning Intent Signals for Mobile DSPs

Show HN: I made a Mac app to centralize and monitor my dev tools

You.com trains on your data by default with no opt out

Better Call Jev

We ported the original Doom to SQL

Changes to Gemini model access and limits

Guillermo Rauch: "We've confirmed a KVM 0day through our Vercel bounty program"

Show HN: Terminal Gym – a Claude mod that makes you do pushups between prompts

An Algorithmic Failure Beneath the Secret Ballot

The KV Cache Is the New Memory Wall

The Server Side Cloud Swift SDK

Holocron: The Backup of Last Resort

Show HN: I made a browser-based instrument out of the CHOMPI CLUB source

How to Create More Than You Think

We asked three AIs about the risks of AI. They disagreed with their makers

13% of my website visitors block Google Analytics

What's the Date?

Using Theme park rides and smartphones for demonstrating Newtonian mechanics

Being a good CPU neighbor (2016)

Show HN: thezoomist.com – Closer to Everything