Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Apple Caught Off Guard by AI Demand for Mac Mini and Mac Studio

Show HN: agix – addresses and messaging for agents working for their humans

Trivium 30 aug 2026

The new Go JSON API: twice as fast, or 1.5x slower? – Daniel Lemire's blog

Dataloupe – turn any CSV, Parquet, or Excel file into an offline HTML explorer

Historical Discoveries That Reshaped Modern Historical Understanding

Guido van Rossum on Parallel Python: "Personally I wish we'd stuck with the GIL"

Show HN: Astral – Astro.js, but in Elixir

Show HN: I built this social leaderboard game

The corporate espionage firm exposing Chinese state actors

The Supabase graph on GitMir could be a CS university course

ArkDisk – managed Nextcloud on hardware I own, in Belgium and Lithuania

Linear models from least squares to quantiles

AI is hurting US politics. Here’s how candidates could use it for good

Two Pixels Become One

Show HN: Our GLM-5.3 Flash Switchless recipe is now out for 4x DGX Sparks

Google Maps switches to "Lake America" as Canada compares change to "SNL" skit

Ask HN: Claude vs. Codex – which one do you prefer, and why?

Catching logs trying to gaslight your security agents

It's not insider trading, it's offshore gambling

Consultants head for a showdown – with their own clients

Nintendo's Exercise Bicycle and the Vagaries of Tech Innovation

I Found the Performance–Cost–Speed Sweet Spot with LLMs

Show HN: Three Slicer, I ported OrcaSlicer to run in the browser

An AI coding agent silently erased 92% of AI nodes in n8n's most-cited dataset

Show HN: SlideOps, slides from a repo that flag when they drift from the code

Show HN: Stickyboard – Kanban on real sticky notes, no account needed

The British state has lost the argument

Show HN: The Economic Atlas: a data-first dashboard for comparing countries

Azriel Blackman, Airline Mechanic for 80 Years, Dies at 100