Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•11mo ago

Comments

nine_k•11mo ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•11mo ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•11mo ago

They mention this in the article.

spockz•11mo ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•11mo ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•11mo ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•11mo ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•11mo ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•11mo ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•11mo ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•11mo ago

This was very well written. Excellent article!

NetOpWibby•11mo ago

Is this like MessagePack for Haskell?

Content Scraping Issue: Risks and Dangers of Artificial Intelligence

Anthropic's Mythos Model Is Being Accessed by Unauthorized Users

The Birth Certificate for AI Agents

Yojam: A macOS default-browser shim that routes URLs through a rule engine

Embedded 191,922 Met artworks to find hidden twins across 4k years

Show HN: CLI-use – turn any MCP server into a CLI in one command

Tenstorrent's Large Compute Cluster, Generates Video Faster Than Real Time

A Boy That Cried Mythos: Verification Is Collapsing Trust in Anthropic

Objectively Measured Daytime Napping Pattern&All-Cause Mortality in Older Adults

Edster – An open-source local AI agent with swarm mode and a web UI

Parrot is a C++ library for fused array operations using CUDA/Thrust

What America Has Lost in the War with Iran

Might be the most Hollywood style launch video I've seen

Terence Tao: On the importance of partial progress (2012)

TSRX – TypeScript Language Extension for Declarative UI

MCPorter – Call MCPs from TypeScript or as CLI

WireGuard for Windows Reaches v1.0

Why WooCommerce Needs an AI Store Manager, Not Another Chatbot?

Canadians pay more than 3.50x than Poles on matched BTC card route

Fontspots: Eurostile

Show HN: I built a coding agent that works with 8k context local models

.NET 10.0.7 Out-of-Band Security Update

Trump is super unpopular. So why don't Democrats have a bigger lead?

AI Unmasked: a series of courses on AI without the hype

We heard you: the new Framework Laptop 13 Pro [video]

Ask HN: Claude Code Alternative

Air Force can't quit the A-10 Warthog, extends service into 2030

Tell HN: My open-source project hit 5k registered users

The distribution of power and inclusiveness across deep time

Taskd – a new task management system