Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Show HN: Trust – Coding Rust like it's 1989

How to Buy Cheap Claude Tokens in China

DNA donors help identify Franklin expedition members

I'm now "writing" my blog

Show HN: StackSense – AI/data/systems engineering knowledge graph

Show HN: Password-less authentication via Ramanujan's 1/π series and Nvidia-B200

MRC Protocol: Supercomputer networking to accelerate large scale AI training

The surprisingly complex journey to text-selectable client-side generated PDFs

Actor alleges James Cameron used her teenage face to create 'Avatar' character

Show HN: CRO analysis tool that finds conversion issues

Show HN: Kstack – Skill pack for monitoring/troubleshooting K8s in Claude Code

Great Attractor

Don't Automate Your Moat: Matching AI Autonomy to Risk and Competitive Stakes

LAWS: A new transform operation turning LLM inference into cheap cache lookups

Elon Musk's Lawyers Ask OpenAI's President Why He Is Worth $30B

Publishers sue Meta, claiming it violated copyrights in training AI with books

Red Button, Blue Button

ABG – An Online Bioinformatics Lab

'FOMO has proven a stronger incentive than poor stock performance' Goldman Sachs

Europe Tech Tools

Prompt API: No web standard should require users to agree an ads company's TOC

GitHub outage: CCR and CCA failing to start for PR comments

ZAYA1-8B: Frontier intelligence density, trained on AMD

Russia's top secret spy school teaching hacking and election meddling

Gephi 0.11 is out: performance improvements

The boring way to build a startup

Openrouter SDK for Elixir

Blink – AI Assistant. A knowledge destination

Citadel Securities: Why C++ wins in finance [video]

Open weights are quietly closing up – and that's a problem