Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•11mo ago

Comments

nine_k•11mo ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•11mo ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•11mo ago

They mention this in the article.

spockz•11mo ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•11mo ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•11mo ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•11mo ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•11mo ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•11mo ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•11mo ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•11mo ago

This was very well written. Excellent article!

NetOpWibby•11mo ago

Is this like MessagePack for Haskell?

Madadh – deterministic fail-closed control system for protected runtimes

Artemis II crew splashes down in Pacific Ocean, ending record-breaking moon flyb

Bring Back Buddy

SBTI Personality Test – The Funniest Personality Quiz You'll Ever Take

Enforcing new limits and retiring Opus 4.6 Fast from Copilot Pro+

An Agent That Grows with You – Hermes Agent

To Fill Air Traffic Controller Shortage, FAA Turns to Gamers

Is VC the new PMF strategy?

AMD GPU LLM Performance Testing

Bringing Rust to the Pixel Baseband

Jonathan Ive, Designer, Apple – "A computer can be sexy."

Sam Altman speaks out after alleged attack on SF home,links to rising AI anxiety

Show HN: Palmier – Dispatch and schedule AI agents from your phone

Latest Episode of Apple's "For All Mankind" Erases Taiwan

We open sourced KubeezCut, a free browser based editor integrated with Kubeez

Framework: ' personal computing as we know it is dead'

'Blips' led Australian backyard astronomers to a world-first planetary discovery

Families Spent Decades on Louisiana's Bayous. The Power Company Pulled the Plug

Relics of an ancient sandstorm on Mars point to Earth-like winds

Yout.com Hopes Supreme Court's Cox Ruling Helps Its Case; RIAA Disagrees

Show HN: I forced Claude to play Tetris in Emacs

Elon of Ed-Tech

Tutorial on Event-based Cameras(2020) [pdf]

New Kodak Film – Verita 200D Colour Negative

The War Is Turning Iran into a Major World Power

Google's Gmail Upgrade Decision: 2B Users Must Act Now

Give your coding agents the context your repo has

Neural Computers

RAG 40x faster using binary quantization (2024)

From error-handling to structured concurrency