Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•8mo ago

Comments

nine_k•8mo ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•8mo ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•8mo ago

They mention this in the article.

spockz•8mo ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•8mo ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•8mo ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•8mo ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•8mo ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•8mo ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•8mo ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•8mo ago

This was very well written. Excellent article!

NetOpWibby•8mo ago

Is this like MessagePack for Haskell?

When Earth Becomes Myth

China's Deepin Linux gets a slick desktop – and, yes, built-in AI

Show HN: Obsidian Workflows with Gemini: Inbox Processing and Task Review

Hang on, there's a Trump Phone Ultra coming too?

Terence Tao: A crowdsourced repository for optimization constants?

Dead Man's Switch: How to Automate Crypto Inheritance

What Happens to Your Crypto When You Die?

We will rewrite SQLite. And we are going all-in (2025)

I Stopped Reading Code. My Code Reviews Got Better

Show HN: Startups.in: An in-development "global" startup intelligence database

Report on European technological sovereignty and digital infrastructure

Interacting with Developers on Reddit

We Hacked TikTok SEO and Got the AI to Recommend Our Product

Tesla discontinues Autopilot to boost adoption of its Full Self-Driving software

Booking App for Freelancers

OpenAI is planning to take a cut of Customers' discoveries

CVE-2026-0915: GNU C Library Fixes a Security Issue Present Since 1996

New YC Site

Show HN: PingLater – A beautiful Chrome extension to snooze tabs

The Enchiridion by Epictetus

Zotero 8

Open Source has a "free rider" problem

TV's are getting bigger and AI inside

Flux2kle.in Fast and Free Image Generator

Learning to Discover at Test Time

Microsoft gave FBI set of BitLocker encryption keys to unlock suspects' laptops

The Gödel Problem: A Mathematical Argument Against AI Thought [video]

Building a product in 20 hours and growing it to a 5-figure ARR

Fighting AI Slop

Submit a pitch: what needs to be built before advanced AI?