Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Open Source libraries are evolving into reference implementations

We put llms.txt on 83 websites. In 12 weeks OpenAI's crawler read it 7 times

Agentic Code Quality

Steam Game Recommender, with a RAG System That Focuses on Explainability First

Fideliya Pass

Anthropic in Talks to Buy World Model AI Startup Decart for $6B

Video game lawyer says all her clients have anti-AI contracts

Lightweight Haar Wavelet Subband Pruning for LLMs

Why your LLM ignores what you asked for

Apple and AI: Siri the Early Years

Shot on iPhone. Now Prove It

When the Denominator Disappears: FDA's Next Evolution Should Be Causal Inference

Show HN: Chatlens – Search and browse your ChatGPT and Claude chats offline

Is Alibaba going back on their words on qwen3.8 27B? this page is now 404

Greatest Concept Albums

The web's newest weapon against AI scrapers is a font

Proofs and Prompts

The Machine Is My Session Band, but It Cannot Hear Me

Stunning Eclipse turns daylight into darkness across Europe

Whoop replacement noop 10.0 released

Show HN: Interspectr – automated interview prep generated from a job posting

ICE plans to equip agents with gloves that can deliver electric shocks

Show HN: A local RAG desktop app: piFlow

NagaAI – Cheaper AI Aggregator (Like Poe, OpenRouter, etc.)

AMD Instinct MI455X Deep Dive: CDNA 5 Marks the Next Era of Instinct

Jarvis – autonomous assistant for real-world tasks

Schema - new language for serialized data in C++/C#/Golang and Rust

Jeff Dean has been in talks for a $10B valuation for his new AI startup

Vercel Sandbox Now Runs on Vercel Managed Images

What Is AI Governance and How to Operationalize It?