Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•9mo ago

Comments

nine_k•9mo ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•9mo ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•9mo ago

They mention this in the article.

spockz•9mo ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•9mo ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•9mo ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•9mo ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•9mo ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•9mo ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•9mo ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•9mo ago

This was very well written. Excellent article!

NetOpWibby•9mo ago

Is this like MessagePack for Haskell?

Apple Plans M5-Based Private Cloud Compute Architecture for Apple Intelligence

Deterministic Core, Agentic Shell

CalyxOS puts privacy and security into the hands of everyday users

ChatGPT's Translation Skills Parallel Most Human Translators

The Broken Equilibrium

Don't Trip[Wire] Yourself: Testing Error Recovery in Zig

What evidence format helps technical diligence decisions move faster?

phyz: Differentiable Physics Engine for Rust

Why I don't think AI is a bubble

AI Panic Hits Trucking, Transport Stocks

Paper, Scissors, Gravity

Opinion: The Finance Industry Is a Grift. Let's Start Treating It That Way

You probably can't trust your password manager if it's compromised

Show HN: Lightweight tmux session and project manager written in Bash

Student

Gave Claude photographic memory for $0.0002/screenshot

After Noam

IBM tripling down on hiring new grads

Focus Better with a Dressing Table with Mirror and Lights

A C compiler in TypeScript, Written by Claude

Show HN: Quackback – Open-source customer feedback your AI agent can triage

Three Sketches of ASL-4 Safety Case Components

Use skills smarter with SkillDock's CLI manager

Index, Count, Offset, Size

I taught LLMs to play Magic: The Gathering against each other

Gentoo Takes the First Step to Ditch Microsoft Copilot-Infested GitHub

I've Created Modular Open Source Shelters

StewReads – Turn Claude chats into Kindle ebooks

The Cloud Above the Clouds

Metadata Is Not Understanding: Knowledge Graph Version Control for AI Code