Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Dissecting Spring petclinic (Part 1)

For Tesla owners – a whole bunch of stats from your vehicle's own telemetry

Show HN: RedHands – MCP for multi-stage offensive security tools

Beating the Frontier on NetHack

RISC-V assembly course – ASM Editor

Building an Agentic Loop in Temporal

Blindsight

I built an arcade within ChatGPT

Herdr for iPhone, iPad and Android

We tested eight new decision models. None got more right than the one we use

df-script – 43kB zero-dependency DataFrame engine for TypeScript and JavaScript

Tensor Seeks Layout: Formalizing Layout Selection for ML Compilers

AI Company Employees Should Unionize

Building a Ethernet Switch ASIC chip

San Francisco weighs limits on self-checkout stations

Show HN: Loft – offline Q&A over your local files, with citations (llama.cpp)

WhatWeRun, a database of real production AI agent stacks

Films, shows, and games to grow up with

Wojak-Bench 3D: Which AI is best at 3D printing?

Rows vs. Columns: How different are they, really?

Show HN: Radiotherapy Delivery Demo Written in Rust and WebAssembly

October Outage: Technical Overview

MergeAtlas: Turn PRs into a Map of Cards

My agent org speedran a company

Tuberculosis rapid test allows in-the-field results

Dartmouth (Guarini) will not admit math PhD cohort this year

Customer success job postings: adoption in 78%, churn in 14%

Proof of Corn (2026)

Lego ZX Spectrum

Show HN: Smart Node, a self-hosted crew of AI agents for a small business