Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Revisiting inferences of trustworthiness from moral judgments

Convert PDF to Markdown and JSON

A Family Tree of Software Job Titles

Tesserae: E-ink dashboard companion. Compose dashboards and push to e-ink panels

Exploration–Exploitation Dilemma

Ask HN: Still Getting [Dead] Posts

Show HN: Builderoom – find indie-made software by describing what you need

Orpheus.el: Emacs Transient Music Player (SSH, MPD, Local, YouTube, etc.)

Runwisp: Cron job manager and process supervisor

Activae iOS app – turns forgotten screenshots into useful actions

Show HN: Indic-Turn-Detector – open-weights end-of-turn model for voice agents

Recursive Functions of Symbolic Expressions and Their Computation by Machine

Extending concurrent separation logic to the hardware level to verify the xv6 OS

Why Did Palantir CEO Alex Karp Buy a Swedish Forest?

Agent Substrate

Show HN: Stop Safari auto zoom on input fields

Mytypefyi

Show HN: Check This Chrome Extension Christian Prayer Journal&Tracker Free

Why Every AI Model Will Ship as a Torrent

Reverse engineering a vintage diode-transistor logic chip

My memory isn't getting worse, it's just using exponential decay

Show HN: Vibe Code 1k

Does It Matter If Men Pee Sitting or Standing? A Randomized Test Has an Answer

API Craft Pro – Chrome SidePanel API Mocker and TypeScript/Zod Generator

Finance team's ChatGPT habit is a cross-border data transfer

Show HN: OpenPocket – iOS app for running Qwen3.5, LFM and other LLMs offline

Show HN: AntFabric (One Subscription for all your operational needs)

Humble Bundle just killed most of their security through vibecoding

A Mad Max-style desert run built from nothing but (procedural) code

GrapheneOS Pixel 11 support is delayed indefinitely