Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

From Isolated Agents to Agentic Mesh: Orchestrating SDLC with A2A and AP2

The Baffling World of Masayoshi Son's Presentations

Research as a Stochastic Decision Process

Self-proclaimed King of Switzerland uses loophole to build his empire for free

Monedula Apache Kafka Simulator

Jonas Lauwiner

California State Government Launches AI Job Loss Tracker as Layoff Fears Grow

New EU rules: military age Ukrainian men to lose refugee visas Jun 27 2027

The Naibbe cipher: a cipher that produces Voynich Manuscript-like ciphertext

I created a new open-source project

AI 2027 Tracker

Alan Greenspan Has Died

How much? The hidden costs of restaurant dishes

Midwit Cleanse – midwits wipe themselves off the gene pool

Translating Pandas to Polars using LLMs

Paris to ban drinking alcohol in public as hospitals hit heatwave breaking point

Paying for LLM inference by the kilowatt-hour instead of per token

How your team can save 100's of hours of work

Google Vids: AI-Powered Video Creator and Editor

GuixPkgs: Every Guix package, as a Nix flake

Micron blames Apple and customers for the lack of memory capacity

Patent-CR: A Dataset for Patent Claim Revision

Building effective pen-testing agents

The Age of the Solopreneur

CasaOS: An open-source home server OS for Docker apps

The Human Agentic Gap

Show HN: Helios – Business OS for Freelancers

What if plants could talk? (OpenAI YouTube) [video]

Snap's Evan Spiegel, Miranda Kerr help erase $550M in med debt for Californians

Show HN: A Transformer Is All You Need