Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•8mo ago

Comments

nine_k•8mo ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•8mo ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•8mo ago

They mention this in the article.

spockz•8mo ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•8mo ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•8mo ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•8mo ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•8mo ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•8mo ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•8mo ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•8mo ago

This was very well written. Excellent article!

NetOpWibby•8mo ago

Is this like MessagePack for Haskell?

A lot could happen in space this year

Golomb-Rice coding for compressing a set of hashes

LLM Memory Is Broken

GLX: A Bash Replacement–Oriented Programming Language for System Scripting

AI gig work explainer [video]

So, you want to serialize a B-Tree?

ICE Is What Happens When America Refuses to Learn from Black History

The Realities of Generative AI in Software Engineering

HEINEKEN's Digital Transformation: Why Change Management Comes First?

Orbital Rocket Simulation

The 1000 Commits Problem

The Value of Technological Progress

Apple's John Ternus Could Be Tim Cook's Successor as CEO

How problematic is resampling audio from 44.1 to 48 kHz?

Ollee Watch one: Drop-in smart PCB for Casio F‑91W

Code Review in the Age of AI

GLX: A New Programming Language, Replacement for Bash and Other Shell

Cloudflare: /cdn-cgi/ Endpoint

If you think you are good at math, you need to change your major out of STEM [video]

Show HN: RCS Composer – a visual editor that outputs RBM JSON

MCP CLI: Dynamic discovering and interacting with MCP servers

A Year of MCP: From Internal Experiment to Industry Standard

Ash HN: Excavating Decision Archaeology

Scroll to Accept? – AI's pull-to-refresh moment

Automatic TLS Certificates for Common Lisp with pure-TLS/acme

You Can't Debug a System by Blaming a Person

Beating the House for the Love of Math

AngelScript

An alternative to code mode: serverless MCP

Meta Unveils Nuclear-Power Plan to Fuel Its AI Ambitions