Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•10mo ago

Comments

nine_k•10mo ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•10mo ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•10mo ago

They mention this in the article.

spockz•10mo ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•10mo ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•10mo ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•10mo ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•10mo ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•10mo ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•10mo ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•10mo ago

This was very well written. Excellent article!

NetOpWibby•10mo ago

Is this like MessagePack for Haskell?

Notes from a Law Professor with No Idea What's Going On

Benchmarking Distilled Language Models for Performance and Efficiency

Designing a Build System That Runs Untrusted Code

Show HN: A complete, containerized data engineering learning platform

Search Quality Assurance with AI as a Judge

Nvidia announces Vera Rubin Space-1 for orbital data centers

Risk Management and Hedging in a Polymarket Dual-Side Arbitrage Bot

Hithium to invest €400M in Spanish mega battery factory

Thariq's Lessons from Building Claude Code: How We Use Skills

Trump's plan to shut down weather and climate center triggers lawsuit

Overseas 'content farms' creating political deepfakes uncovered

Show HN: StackStats – Analytics tool for Substack writers, runs 100% locally

Et tu, S&P 500? The SpaceX IPO gamesmanship is going to be epic

You're Not Thinking About Your Network the Way You Should

Did Cinema Get Narrower?

Turning raw logs into feature vectors without manual labeling

Show HN: Starting Five – NBA Lineup Building Challenges

SecOps without manual schemas: Using EBMs and automated ETL for detection

Ban Bots Not Human Directed Tool Use

Show HN: Horizon – GPU-accelerated infinite-canvas terminal in Rust

Fair Source Software in the AI Age

AI Agents and the New SaaS

YouTube is experimenting with ads visible even after users skip

Stop training your security ML on labeled attack data

Why does it feel uncomfortable to think about how much you use your phone?

Stripe.com/6oU7sL9Pwg6Xa9kBest AI Agent Certi1iK1gs0s

Spectra – detect API contract drift from real runtime traffic

What was DOGE? How Elon Musk tried to gamify government

Why Claude Code Can't Find Your Tools

India's outsourcing industry is worth $300B. Can it survive AI?