Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•12mo ago

Comments

nine_k•12mo ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•12mo ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•12mo ago

They mention this in the article.

spockz•12mo ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•12mo ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•12mo ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•12mo ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•12mo ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•12mo ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•12mo ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•12mo ago

This was very well written. Excellent article!

NetOpWibby•12mo ago

Is this like MessagePack for Haskell?

Show HN: Figma alternative where AI works with vector primitives, not code

Is Email Confidential in Transit Yet?

Does Point Cloud Boost Spatial Reasoning of Large Language Models?

Chip Startup Aims to Shatter AI's Dreaded Memory Wall

Mild dehydration affects mood in healthy young women

Scaling Test-Time Compute for Agentic Coding

Show HN: Rocky – Rust SQL engine with branches, replay, column lineage

Workday's Last Workday

Big AI cluster little power the 8x Nvidia GB10 cluster

The Monetization of Spam

The Making of Digital Identity: Mobile Revolution and the Surveillance Machine

The Pragmatic Prompter

Show HN: Keychase – A zero-config, offline Python secret scanner

GPT-5.5 prompt for Codex tries to make it not talk about goblins

I Ditched the iCloud Keychain

Scammers use Gmail dot alias trick to spoof Robinhood in phishing scam

OpenAI's revenue, growth estimates fall short as company races toward IPO

Show HN: Integrations gateway for agents with 2FA for destructive ops (OSS)

U.S. is 'being humiliated by Iran,' says Germany's Merz

Free retirement tax calculators – SS tax torpedo, IRMAA cliffs, Roth ladders

An AI prompt-injected another AI in the wild and recognized it had succeeded

Wysiwyg LaTeX Editor Compositor for Windows and macOS Release 0.7

Open grid data has a public benefit

PostgreSQL and the OOM Killer: Why We Use Strict Memory Overcommit

Clojure on Fennel part three: parsing

Understanding Systems

Proton VPN's promises post-quantum groundwork, Stealth for Linux and new apps

Where Optimizations Come From

Start with the sensors, then design the rest: How Zoox built its robotaxi

Microsoft Ecommerce Platform Options for Scalable Online Commerce