Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

America, 1926: What a Forgotten 100-Year-Old Report Says About Who We Are

Why are we still uploading PDFs just to compress them?

Best Simple System for Now

Show HN: AI latent space with overlapping manifolds

The Honest SendGrid Inbound Parse Alternative – MailKite

Jamesob's guide to running SOTA LLMs locally

It Still Can't Do My Job: Four Years of Moving Goalposts (2022–2026)

The and Justice for All" Book Club

Battery startups see 'crazy' demand to smooth power surges in data centers

Dropway: Share LLM artifacts with your team

Show HN: SigRank – Competitive Stat Screen and Operator Performance Evals O7

Golden Paths Weren't Built for Agents

AI coding is addictive. Engineers are paying the price

Mistral vs. Claude on our onboarding: 4× faster, 30% cheaper

How Fighter Jets Lock on (and How the Targets Know) (2014)

Give Smart People the Tools to Do Smart Things

Well, the Steam Machine was pretty cool for the 20 minutes that it worked

Pilot Shell: Spec-driven plans; enforced quality gates; persistent knowledge

Stop AI from Wrecking Your Codebase with Spec-Driven Development

Building an Intern

Show HN: LimitBar, a macOS menu bar app showing your Claude usage limits

Dad Books Are a Dying Breed

An E Ink ESP32-Based Game Boy

Trump portfolio tracker account suspended one day after launch

Using OpenTofu's Exclude Flag to Isolate Performance Bottlenecks

Show HN: Hello Inbox – Maximize inbox placement for email marketing campaigns

Build a blog engine in 15 minutes with Ruby on Rails (2005) [video]

C++ Under Pressure: A Practical Interview Guide from 1k Technical Interviews

Colophons, and why I'd never heard the word

Show HN: Finding better quantum error correction codes using ILP