Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Waymo to Deploy Robotaxi with Zeekr to Expand Public Rides

New Retype for Obsidian Plugin

A company turned visas to Europe into big business

One week later, the same AWS t3.large VM changed personalities again

Ask HN: What Do You Struggle With?

International Masturbation Month

I built an Android-like OS that runs in the browser

OpenClaw Architecture – Part 1: Control Plane, Sessions, and the Event Loop

Nuclear Space Power and Propulsion [video]

Real-time LLM Inference on Standard GPUs (3k tokens/s per request)

Deep Tech Companies Are Built Different

The Download: climate tech goes public and the AI Hype Index returns

Leibniz's Explanation of binary arithmetic (1703)

What the Pope said about AI

Cities Are Covering Flock Cameras with Trash Bags

The AI Productivity Trap

Linux is a dumpster fire – somebody call the fire department

Tame your pesky little scripts

Large Language Models in Software Security Analysis

Dynamic Workflows in Claude Code

Un asistente de IA que termina tareas reales en tu ordenador

Claude Opus 4.8

Beyond the Prompt: Claude Code

Show HN: Timelocked – local, offline timed-release file encryption

Tracing HTTP Requests with Go's net/HTTP/httptrace

Tuning LLVM's SLP Vectorizer Cost Model

The Agent Trust Stack: A Layered Framework

Trivial Pursuits

US and Iranian negotiators reach tentative deal to launch nuclear talks

Chatbots' Downward Spiral