Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Reverse Engineering the Apple QuickTake 200 in 2026

Flock license plate reader wrongly linked a San Diego man to a violent crime

Show HN: Luminous – fast image viewer in Rust, SAM 3 and CLIP support

MemGraphRAG: Memory-Based Multi-Agent System for Graph RAG

Raspberry Pi Home Server: Back Up Your Files Locally for Under $100 DescriçãO

Dead Button Syndrome

Boomers are hoarding most of America's wealth and power

What Are Tokens in LLMs?

Jorge Luis Borges Lecture April 9th, 1976 [video]

Criblate – Help your favorite creators speak every language

The Tinker Pledge

Xaml.io Compiles to Native Apps in the Browser

Easter egg in new Lego Batman game contains working C64 code [video]

The architecture of the internet creates risks for democracy

Show HN: Hardbar – compile-time defined i3bar

Ask HN: Job market for SDMs/Engineering Managers. Any reliable data?

Loss of Prefrontal Cortical Higher Cognition with Uncontrollable Stress (2019)

Show HN: Nightwatch, The open-source, read-only AI SRE

Mesoscale carbon fiber lattices with foam-like weight and bulk strength

Behind Every Dad Bod Is a Healthy Dad Brain

Modular morals: Mapping the organization of the moral brain (2024)

I built a Discord status for Claude Code

1982 World Championship Double Decker Bus Racing (1982) [video]

Try out my version of 1M checkboxes

Agentic AI solved coding and exposed every other problem in SE

China has approved the first invasive brain-computer chip

We May Be Entering a Second Axial Age

AI friends too cheap to meter

Oxford Uni student data pwned yet again, this time via career platform breach

Show HN: A List of AI Neolabs