Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

40 Years Ago, Oceanographers Found the Wreckage of the RMS Titanic

Why Short AI Coding Prompts Can Cost You More Time

Io_uring I/O Can Outlive a Reaped Process

Show HN: A website anyone can edit through auto-merged GitHub PRs

New York Times and The Athletic workers demand company scrap Kalshi deal

A Transformer Plays Tic-Tac-Toe

Justifying text by hand, now at 35 columns

RSA-260 Factored

Astronomers Detect a 10-Sided Structure in Saturn's Atmosphere

Fuck Off as a Service (Foaas)

Show HN: A newsletter about systems performance, scalability, and efficiency

AI could have written this post

Sitelo – Static Site Generator

Coloring the Snub Dodecahedron

Active Inference: The Free Energy Principle in Mind, Brain, and Behavior

Show HN: DumbDetector – Community signals for AI tool problems

ARKM KERNEL approaching you soon

Ask HN: Is SpaceXAI taking Claude down too?

Balrogg: Demonically compacting (up to 15%) lossless Vorbis/Opus recompressor

The Cognitive Revolution (When Machines Do the Thinking)

ICE Has a $2M Contract for Spyware That Can Hack Phones Without a Click

Show HN: Litelink – local-first, embedded stream capture into Iceberg tables

Raycast now requires a subscription for BYOK and local models

Show HN: Pangolin – SSO and WireGuard instead of API keys for LLM access

Life Is Bigger Than GPUs

Paid Phones

Vibe Coding Does Not Have One Definition, It Has One per Developer

An AI agent sliced through an enterprise defenses in 10 hours

21st Revision of Patches for Getting Linux to Shutdown Devices Asynchronously

Live Atomic Spatial Audio