Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

US bank places trust in ransomware crew that promised to delete its data

Entertainment Breakdown: Medium's Surging Pop-Culture Publication

Freedom of Thought: The Fast-Growing Medium Publication That's Here to Stay

Ask HN: Am I the only one getting so many cursor ads?

JSON Feed Version 1.1

Ok that's a better RPi project – Lego submarine [video]

Typewriter Art [video]

Estonia, the Country of the Fit and the Wit

Falling in and Out of Love with Esperanto

RowLoom – clean up messy CSVs, no signup for a one-off file

EVE Online but for Hermes and OpenClaw agents: MMORPG your agents can play

Hawaii's Bake Stands

Show HN: Bor – Open-source policy management for Linux desktops

I Built Handysuite as a Native Apple App (and Skipped Cross-Platform)

AirLLM: Inference 2.8T Kimi K3 on a single 4GB GPU

Ask HN: When will ChatGPT Voice be able to control my phone?

A daily estimation game where the crowd's answer fights an AI

The time filter in Google Search broke a few days ago

AGI Singer: Buddy, I Need an AGI Handover Framework

EU Icons for labelling AI-generated content – Shaping Europe's digital future

Belladonna of Sadness (1973)

Tesla Spyware

Don't replicate data over webhooks. SCROLL

POV: An AI replaced your job this morning [video]

The Claude Code plugin marketplace has no price field

What Happens When Frustrated Machines Talk to Each Other?

Running a 35B LLM at 128K Context, Full Speed, on €870 of Used Hardware

Adima AI: The Future of AI Enhancement

Every new UK home should have air conditioning, say Liberal Democrats

Ginger hair portraits show unexpected 'golden thread' connecting Americans