Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•9mo ago

Comments

nine_k•9mo ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•9mo ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•9mo ago

They mention this in the article.

spockz•9mo ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•9mo ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•9mo ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•9mo ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•9mo ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•9mo ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•9mo ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•9mo ago

This was very well written. Excellent article!

NetOpWibby•9mo ago

Is this like MessagePack for Haskell?

Aside from That, Mr. Cook, What Did You Think of the Movie?

"Agentic CI – CI that knows which tests to run (LLM-powered)"

American horror story: Early issues for TikTok US

Rise of the Supergrapes

Turn any image into an emoji mosaic

Wdlist: Create and manage a personal list of Wikidata items

If You Tax Them, Will They Leave?

Whats the Fuss about Clawdbot

Ariel OS – a Rust RTOS for IoT Microcontrollers

Show HN: FavBox – local-first bookmark manager extension

As we breach 1.5°C, we must replace temperature limits with clean-energy targets

Show HN: I made a list of validated mobile app ideas

UEFIGame: "Win –> Boot, Lose –> Shutdown"

Show HN: Flashy – Path-based reverse proxy for routing multiple services

Show HN: Klynda, a portfolio tracker focused on true performance (TWR/XIRR)

ICE and Palantir: US agents using health data to hunt illegal immigrants

Kimi K2.5 on 2 Mac Studios (<$20K) running at 24 TPS

The World's First Nintendo Switch 2 Pro Controller with TMR Glide Sticks

Australia: Telstra warns older iPhones may be unable to connect to Triple Zero

Show HN: PolyMCP – Expose Python/TS functions as MCP tools easily

Amazon laying off about 16,000 corporate workers in latest anti-bureaucracy push

AI Kind of Sucks at Retouching, Study Says

Claude Code Tips for Large Projects

Show HN: Camera Benchmark Tool – Fast, Clean Side-by-Side Camera Comparisons

Show HN: SHDL – A Minimal Hardware Description Language Built from Logic Gates

Wire Messenger Is Down

Particle is being acquired by Digi to power the next 40 years of IoT innovation

Introducing: React Best Practices

Building a JavaScript runtime in one month

The SQL Query Roadtrip: The Indexes