Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Big Pharma subsidizes biotech investors

Show HN: Just Dashboard, an open-source VPS management dashboard

MAI-Transcribe-2: Fastest, most accurate and cheapest speech recognition model

Venture Multiples and 10 Year Cycles

Use Uncensored Models on OpenCode, Codex, and Claude Code

AMD implements new subpar swap system

ExxonMobil launches legal challenge to EU carbon capture rules

Adobe Source Libraries Overview (2007)

Reconsidering the drivers of ice-shelf collapse

37% on ARC-AGI-2 Kaggle Without LLMs

Cloudflare Clairic: a single pass extraction model

Patio11 bribes State Department official

The 31-Year-Old Who Called Out Jensen Huang's 'China Problem'–and Shorted Nvidia

OpenAI Fooled by Russian Pranksters into Thinking They Were Talking to Ukrainian

Just say no to Anthropic's IPO

Epigenetic heterogeneity in breast tumours informs diagnosis and prognosis

Reddit broken on desktop as full-screen mobile app prompt takes over page

ApoB-100 – 50 years unsolved, measured disordered, folded to order

Wiki.js 3.0 Release

Lego marketplace BrickLink offline after 'unusual activity' detected

Jobdork – job leads from employers' career sites, ranked by your resume

Coding is dead, long live Coding

U.S. Partner

What gives you the chills? AI can engineer it

San Francisco youth culture died at a peptide party in Fisherman's Wharf

Yes, you should panic about AI

Lessons from 59 Linux Kernel Bug Reports

Keeping names, emails and phone numbers out of LLM prompts with a local proxy

Porting my Electron app to Rust: 675 MB to 91 MB

Show HN: Tsfm-pg: Run Chronos-2/Timestral forecasts inside Postgres with SQL