Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

AI Marketing

The teenage hitmen recruited to kill across Europe

Futurism says Meta has "almost nothing" to show for AI. The numbers say

Ask HN: Do Something about Bots

MkLinux and the pimped-out Apple Workgroup Server 9150

Linux desktop market share has hit over 10% in North America

Don't Take the Black Pill (Text Adaptation)

Show HN: FirstEmployee.ai – Start a Business

Plug-in solar is coming. Plug-in batteries should follow

Deep-sea vehicles spot 'alien' sharks deep beneath the waves in the Pacific

The world first agentic AI radio

Recurse Center Halftime Reflection

Four Time Scales for Technology Development and Deployment

Design Thinking Is B.S. (2018)

DeepMind Disbands AlphaFold Team, Pivots to Gemini

Building agents that survive their own execution

When random.bytes() runs but doesn't work

Show HN: I'm a 15 Year Old Wannabe Engineer, This Is a Cycloidal Gearbox I Built

What 'Pluribus' Gets Right About the Hive Mind (2025)

Portal: Current Events

Fuck the Algorithm. Own Your Data

Living in the Math World Means Threading the Needle

Show HN: CostPerPrompt – Live AI API pricing and real-workload cost calculators

Inverseminar

'Death by organ donation' floated for patients approved for euthanasia

Hiring FDE's (full stack software eng) // 100K+ & equity // AI for Manufacturing

I made a free directory to submit your games

Token diplomacy: How China is shaping the AI future

Bessent and the Fed help Japan reverse months of yen losses

The Linux distro and desktop brawler