Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•8mo ago

Comments

nine_k•8mo ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•8mo ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•8mo ago

They mention this in the article.

spockz•8mo ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•8mo ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•8mo ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•8mo ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•8mo ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•8mo ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•8mo ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•8mo ago

This was very well written. Excellent article!

NetOpWibby•8mo ago

Is this like MessagePack for Haskell?

Behind Trump vs. Powell Is a Battle over US Empire's Future

It Can Apply and Positive in Favor the Newton III Law on an Engine System Device

State Ofthe Art Novel InFlow 1Gearturbine/Reaction 2Imploturbocompressor/Impulse

San Francisco to offer free childcare to people making up to $230000

Podcasting Could Use a Good Asteroid

Ask HN: What are Claude's skills/what skills does Claude possess?

Glyphhanger – Your web font utility belt

The Myth of the ThinkPad

Jeff Bezos Needs to Speak Up

Ericsson Silent Layoffs in the US

Trump Moves to Make Tech Giants Pay for Surging Power Costs

America's Throwaway Spies: How the CIA Failed Iranian Informants in Tehran

Mark Carney and Xi Jinping meet to mend ties as Donald Trump disrupts globe

Fontello – Combine icon webfonts for your own project

Is there any way we can help Stack Overflow Website get back up?

AI as a Compression Problem

PanoptiCity – interactive map reveals the scale of mass surveillance worldwide

How Safe Is the Rust Ecosystem? A Deep Dive into Crates.io

Trump accepts Nobel Peace medal from Venezuelan opposition leader

Gen X and Millennials Will Inherit Trillions in Real Estate over the Next Decade

From AI agent prototype to product: Lessons from building AWS DevOps Agent

TranslateGemma: A new suite of open translation models

Show HN: Buildzr: Python DSL for Authoring C4 Models

Apple's Tactics Could Prevent Japan from Improving Browser Competition

Boeing knew of flaw in part linked to UPS plane crash

Microsoft Xbox Manufacturing in 2002

Image FX – Free One-Click AI Photo Editor and Image Generator

European Alternatives for Digital Products

Show HN: Dev Utility Hub – Client-side only developer tools (JSON, JWT, Cron)

vLLM-MLX – Run LLMs on Mac at 464 tok/s