Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Ask HN: What do you ask AI to do with your files?

Show HN: Estimania – Estimation Game

Payloads used to dictate the terms of launch. That's changing

Altman: GPT-5.6 is 54% more token efficient on agentic coding

Venture investment smashes an annual record with half a year left to go

AI Agent Platforms Are Getting Hacked. Here's What's Missing

US home prices hit an all-time high as sales slow and mortgage rates rise

Don't Call Yourself a Programmer, and Other Career Advice (2011)

AI Agent Skill Traceability in UX Design

New open access book on history of computers and politics

BareMetalRT – TensorRT-LLM running natively on Windows (no WSL)

PostHog Open Sourced

No leap second will be introduced at the end of December 2026

Maxwell's Equations Were Discovered [video]

Show HN: HyperSwitch – macOS app switching with direct keyboard shortcuts

Show HN: I replaced Whisper with Parakeet TDT on a $55/month CPU server

Chinese hackers reverse engineer NVIDIA GPU to work on a custom PCB with NVLink

Transparency efforts behind the Helium Browser

Show HN: EPEE, an expert-annotated ASL dataset from native Deaf signers

Show HN: ContextNest versioned, governed context for AI agents (open-source CLI)

Show HN: Tokenstead, find AI models for your hardware

Introducing Muse Spark 1.1

I Did Not Kill Stanley Lieber: How to Draw (With 9front)

Pipeline Parallel Decompression

A Startup That Builds AI Agents Used One to Raise $100M

Catnip lotion as effective as Deet at repelling mosquitoes, study finds

Ollama raises $65M, grows to nearly 9M users

Show HN: Gladia CLI: transcribe audio from your terminal in one command

Show HN: EvenKeel – a free financial planning chatbot

Ollama Raises $65M to Accelerate Open Models