Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Tesla claims Cybercab driving employees at Giga Texas – in a parking lot

Mitigating Factual Hallucination in LRMs via Mixed-Mode Advantage Regularization

AI Fuels Startup Boom Across the US

Apple's failed self-driving car program left a legacy of powerful AI chips

GPT-5.6, Fable 5, and Grok 4.5 rebuild Basecamp from the same spec

The Battery Startup Taking on China's Giants

The Trade in Looted Antiquities Endures for One Reason: Demand

What Happened to Behaviorism?

Stop Pretending That Intensive Parenting Doesn't Work

Website is served from a 200KB binary

The Idols of Acceleration

LARP – Revenue infrastructure for serious founders

Show HN: Shirei, cross-platform GUI framework in native Go

Automation Without Understanding

OpenAI's AI Beating Every Human at AtCoder

Blog-Doc Reloaded

Noto: A Typeface for the World

AI and the Future of Writing-roundtable of authors discuss ramifications for art

Apple sues OpenAI over alleged trade secret theft

Britain's biggest community solar farm forced to shut over grid overload fears

Using AI to Let History Speak About Bank Runs

'This was a righteous case. A holy war': the lawyer who took on Meta and Google

Show HN: Broll – an MCP server that gives coding agents a content studio

Show HN: A lightweight app to let LLM work for oncall

The Universal Scalability Law (USL)

Show HN: The Line of Hope – how current teams compare to past champions

Honeyprompt: LLM-first honeypot that talks back

How to Start a Peptide Testing Company and Costs Involved

Show HN: Open-source browser mini-game where every level is provably solvable

I connected my net worth tracker to Claude via MCP