Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Show HN: AgentReel – Clips. One Cut

Survival LLM

The Limits of Fingerprint Blocklists: A Reproducible Java Certificate Bypass

What is bigger than AI?

Ask HN: Why Is Anthropic's Customer Support Bad?

Rest of World faces prospect of shutting down

Bitter Lessoned in Automated Theorem Proving

Bootstrapped and Need AWS

The Future of Software Is No Code (2018)

The Paradox of Contemporary AI: Engineering Success and Institutional Failure

Are universities on the brink of an Uber-like revolution?

I Believe P=NP

What's Math Anyway

When will ME be cured?

The Goose Programming Language

Multics – Indieweb Search Engine

Cross-request KV cache exposure in the GLM-5.3 Workers AI model (155 responses)

OpenClaw creator says he spent $10M in tokens last month

Ruliobiology and the theoretical foundations of biological form

.NET October 2026 Out-of-Band Security Updates

Born App: Synthetic Friends

The Small Web is Huge [video]

A software-coded view of the world

The Personal Agent Protocol Map

Is PHP Running on Your ColdFusion Server? How to Check, and Remove It

Information for Python users regarding eMudhra / emSign CA certificate security

Rhode Island sues Meta over alleged profits from scam ads

Who lost my lap infant? My attempt to reverse engineer a decade-long airline bug

A steerable motivation concept influencing persistence in Qwen-3.5-27B

Toolflation: Seller Agents Distort Tool Markets