Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Where U.S. Salary Ranges Help Job Seekers

Show HN: A leaderboard of most popular open-weight LLMs

Catbath – a tiny text editor in Rust with edit over SSH and web editor in 456KB

AI was supposed to make smarter decisions. Iit may be making leaders dangerous

Lianda and the Long March

LineShine Debuts at No. 1 as the TOP500 Enters a New Global Exascale Era

US AI stock sell-off shakes markets from Wall Street to Asia

Want to feel happier at work? Take a five-minute walk

Sakana Fugu: a multi-agent system delivered as one model

US presses Meta to agree to AI reviews as security fears rise

Can a $1,500 Battery Replace a Powerwall? [video]

BenchPress: Predict any LLM's score on any benchmark

Legibility of Effort

Raspberry Pi Pico W as USB Wi-Fi Adapter

Ions, a distributed reasoning graph built from evidence backed claims

I built an open source VAD that beats Silero, Pyannote, and WebRTC

Technology-Driven Moral Panics

Mythos model found vulnerabilities in classified US Government systems

Ten years on, Brexit's economic impact is becoming clearer

RRB-Trees: Efficient Immutable Vectors

2026: The year of the node based editor

Congress Clears Housing Bill, Cementing a Rare Bipartisan Feat

"Fix" MacBook Neo Cursor Lag: Record 1 Pixel of the Screen Every 10 Seconds

InSight: Self-Guided Skill Acquisition via Steerable VLAs

Kevin Warsh's Press Conference Collides into 30 Years of Michael Woodford

The Teensy Executable Revisited

How Quake Ruined Id Software

Bash Line Editor: a command line editor written in pure Bash

The war on terror primed America for autocracy

Show HN: Procman, a TUI for run Procfile based app locally