Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Claude Ecosystem

GPT-5.6 Preview System Card

A German AI publisher rewrites Hacker News posts and strips the sources

GitHub DeepSeek-AI/DeepSpec

A Farmer Arrested for Going 5 Seconds over His Time Limit at Data Center Meeting

Recursive self improvement for human skills

DIY your own open-source Robot Vacuum

Voting with Feet

Videogenr.com

Open handoff: Thought Tree, a markup/spec idea for modular LLM workflows

Quora and mass AI poisoning: An organized crime AI spam ring

US Government allows Anthropic limited release of Mythos/Fable models

United Auto Workers Vote to Divest from Israel in Historic Victory – Truthout

Show HN: DiagramIDE – Diagrams as Source Files

Ask HN: Why did you learn Chinese?

macOS Tahoe update breaks using external display in clamshell mode

More than a month into DR Congo Ebola outbreak, doctors warn epidemic will last

Show HN: As Music Theory – Test your sheet music sight reading speed

'Aren't the Organs a Silver Lining?'

2026 April Unemployment Rate by US Metropolitan Statistical Area

FieldStation42: Broadcast and Cable TV Simulator

The Intergalactic Fermi Problem

Naples Outpaces Milan as Southern Italy Drives Economic Growth

Wikipedia's Cofounder Has Been Banned from Wikipedia

Show HN: BootProof – the run button that can't lie (boots any repo, signs proof)

EU Commission HQ forced to shut down air-conditioning amid heatwave

Spinel on Rails

Multifolio – Take Back Your Phone

Code Is Free Speech

We took all of GitLab and turned it into a Graph DB