Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Admin of darknet market that shut down in 2019 arrested after buying gold bars

Building audit logging in Postgres without triggers

Why Distributed Training Is Hard: DTensor and the Costs of Abstraction

From Open Source Software to Open Source Strategy

Displaying a crafted file in Tabby terminal can run shell commands

Your Most Improbable Life

We built the open-source layer for local AI agent visibility

New York to tax luxury second homes in NYC

Pith

loopmaster – Livecoding Music IDE

The cellphone that implicates Argentina's president in the $Libra case

All Polymarket temperature markets are missing, along with the money bet on them

Agora-1: The Multi-Agent World Model

Migrating from Go to Rust

Grief in the AI Age

Musk Loses Case Against OpenAI

Elon Musk Loses Landmark Lawsuit Against OpenAI

A Markdown-based test suite

Geblang – I built my own programming language

An educational generic architecture (RISC-V, z80) simulator

The Dark Side of the Jevons Paradox

Iosevka + Berkeley = Ioskeley

Show HN: Gambling/math/typing for screen time (iOS)

No big deal or the end of the world?

Haiku OS runs on M1 Macs now

10Gb/s Ethernet: using mini-heatsinks with a 10GBASE-T SFP+ module

Andrew Huberman's Peptide Recommendations 2026

Simulating Infinity in Conway's Game of Life with Modern C++

Show HN: StrudelBot – 3D printed humanoid robotic hand with teleoperation glove

Elon Musk losses OpenAI lawsuit as jury sides with Sam Altman and Greg Brockman