Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Post-mortem after Namecheap/RadiusDC service outage

Towards a Risk Assessment of Malicious Skill Files in Coding Agents

Talos AI Agent Super Secure

Learning Augmented Heuristics

Brands Suddenly Care About Reddit. Redditors Don't Return the Feeling.

Qwen 3.8 27B 4-bit designs an animated dolphin riding a bicycle SVG

How the World Was Wired

Show HN: Macsomnia – close your MacBook lid while keeping long jobs alive

The case for overhauling American science

Show HN: Kvcachescope – Why Nvidia-smi is blind to vLLM KV cache leaks

A Framework for Solving the AI Data Center Energy Crisis

Selling my production-ready 4-in-1 FastAPI AI Automation Suite ($8k)

Production-Ready FastAPI Back End Suite – Advanced RAG and SEO Automation ($8k)

Backtesting Congress members stock trades by the disclosure date

Coding Magic the Gathering in C# – Part One – Why Build a Card Game in WinForms

OT Manual Operations Plan: What Manual Operation Buys You

Migraine Trail: Voice Migraine Tracker

Claude Fable 5 Having Fun

Show HN: A prompt to check Supabase DB security

Where the Shadow Fell

France's Top Court Strikes Down Ban on Social Media for Children

We got content-defined chunking from 555 MB/s to 21 GB/s

Digital Signal Processing Pioneer Bede Liu Dies at 91

Stop sending me PRs; a rant

Musk assembled a full-stack AI coding play while everyone watched benchmarks

Substack forces authors to use Pangram

1973 Concorde Eclipse Flight

Show HN: Another Markdown editor, but this one is a web app

RISC-V: They should have known better

Two slicks appear in Gulf as oil spill off Oman threatens disaster