Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

I find AI roleplay therapeutic

Show HN: Levee – a self-tuning circuit breaker and concurrency limiter for Go

RTX 5070 Ti throttles at 107C; Blackwell hotspot sensor readable via MODS tool

Rep. Ro Khanna was detained by Israeli settlers in occupied West Bank

Flock Camera Conspiracy, Prepare for Whats Coming [video]

Frequently Asked Questions on Expertise

What xAI's Grok Build CLI Actually Sends to xAI

Like a cheat code for your car: We investigate ECU tuning

Apple Hide My Email bug, possibly related to disclosure vulnerability

Firefox 12.58% for Desktop Browser Market Share in North America June 2026

A Erlang style pure Scheme Webserver and further

Dismissive Dan's Review of the Overplane AI Coding Harness

Learning-to-Optimize via Deep Unfolded Flows

Hitting $125k MRR as a solo founder by doubling down on the right segment

A pure scheme web programming tool

A dock that wakes up reliably

WLV-01, a monochrome digital photography camera built from scratch

The biggest steam locomotive is whistle-stopping across the U.S.

"How 500 HN users crashed my 2GB server in 60 seconds – postmortem

Autopsy Study Finds Replicating SARS-CoV-2 in the Hearts of Long Covid

I made a horror game that's IMPOSSIBLE to SCREENSHOT [video]

Social.Wiki is a wiki for editing social web pages

Long Covid May Physically Damage the Nerves That Control the Stomach

Search for isRecord on any open source codebase

Martha Lillard, last US polio patient using iron lung, dies at 78 in Oklahoma

We built a free CVE and threat-Intel command center – no login, no paywall

What happens between entering the prompt and seeing the first word appear

GC shape stenciling in Go generics

Old Icons

El Nino forms, expected to strengthen, say NOAA forecasters