Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Tales O' Tech: Suffering Fools

Can you solve the three cloaks enigma?

Don't Be Fooled by this Summer of AI Hype

Si Sheppard – How did a few hundred Spanish soldiers topple two empires?

Tech in cars can be used to snoop on you, Dutch spy chiefs warn

Show HN: Graphene – Data analysis toolkit for your coding agent

The Note – A Markdown notebook with runnable code and formula tables

Crafting Express – isolated local environments for AI coding agents

Slot Machine Programming and the Hidden Curriculum

Show HN: A 100% Client-Side HAR Analyzer That Parses Logs in Seconds

Sampling to Reinforce

CSS Bed: Classless CSS themes to use as starting points in web development

Hermes ChatGPT Extension – Your Hermes Agents Inside Codex/ChatGPT

Show HN: DOM-defense – turn your current webpage into a tower defense level

Ask HN: What are you working on? (October 2026)

Saturn 1 (SA-5) Camera Inside Kerosene Tank During Launch [video]

Micronaut Framework 5.2.0 Released

Monitoring my Samsung smart washing machine without a cloud

The Specter of Neuralese

Pretraining Latent Information Feedback Transformers with Teacher Supervision

Researchers can reconstruct what you're looking at from a brain scan

Reducing the cognitive load of AI changes

Rust 1.99.0

The death of web development education

Single CPU Language Modeling Benchmark Shattered

Show HN: Fumble – keybr, but it learns from everything you type (macOS)

Who Remembers Etch a Sketch?

Claude.dev Blog / technical writing for people building with Claude

AI as Normal Technology

Vue – A WebAssembly Port of TeXmacs