Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

After 100 years of BBC LW, last ever R4 Shipping Forecast and final close down [video]

McNUFFT – Nonuniform FFT for Apple Silicon GPUs via MLX

Defining Taste

SVG-Margin: Better Gutters for Emacs

Horsewood ComplaintS (2026) We Tried It My Honest ReviewS

All Logic, No Bite

When a measure stops helping people see and starts teaching them to hide

After years of working with Go, I wrote the interview guide I wish I'd had

Australia plans to strengthen under-16s social media ban

Apple is skipping M6 Pro and Max to focus on M7 chips

Democrats and Republicans agree: AI is scary

Mathematics: Its Content, Methods and Meaning

Ford execs say they made a mistake when they replaced human engineers with AI

We measured whether AI obeys architecture rules. Even Opus ignored them 60%

Show HN: OpenClaw Launch – deploy a managed OpenClaw AI agent in 30s

CraftsmanSHIP. Not CraftsmanSHIT

LFM2 VL WebGPU

Qwen3.5 WebGPU

Show HN: An LLM that designs M5Stack hardware builds

Explore the Leading Global Data Center Database

AirPlay for Google (Android) TV Chromecast

Emergent Geometry from the IKKT Matrix Model: Convergence to S^4

What Is an AI Intelligence Layer for Business Data?

Voxel Collision in SuperSplat

Apple asks Trump admin to approve Chinese RAM after product price increases

Blue print to let machines think like humans

Sony's State of Play Showed That Every Publisher Is Terrified of GTA 6

Ping pong: A game that requires a C compiler to play

IBM MCGA Gate Array Reverse Engineering

Testing 67 Models: Combining LLMs Rarely Beats the Best Single Model