Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•11mo ago

Comments

nine_k•11mo ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•11mo ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•11mo ago

They mention this in the article.

spockz•11mo ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•11mo ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•11mo ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•11mo ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•11mo ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•11mo ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•11mo ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•11mo ago

This was very well written. Excellent article!

NetOpWibby•11mo ago

Is this like MessagePack for Haskell?

The Video Software That Powers the Internet (and Almost Nobody Knows It Exists)

Tapes: transparent AI agent telemetry

A foundation model of vision, audition, and language for in-silico neuroscience

How to Kill a Tech Monopoly

Beijing's surprise intervention on Meta's Manus rattles tech founders

Clawbolt: AI assistant for contractors, not knowledge workers

Show HN: A votable list of the one thousand best books

Show HN: Foundry: a Markdown-first CMS written in Go

Jaybird 6.0.5 Released

Speculative Decoding: Performance or Illusion?

Ask HN: Mercury Bank closed account due to residency – alternatives?

LLM Persuasion Benchmark: Multi-Turn Persuasion Between Models

Confinia – a supervised market research runtime for live trading systems

Scaling Laws: Should AI Laws Be Subject to a Higher Standard? [video]

Katpack.ai – a pack of AI agents that argue, vote, and sign before any trade

AI contract review for people afraid to sign

Why Software Engineering Will Never Die Revisited in the Age of Spec Driven Dev

Cache Energy's Thermal Battery

How Apple became Apple: The definitive oral history of company's earliest days

Myelin – Procedural memory for AI agents

The first deterministic RAG system (same query = same results, every time)

Browser-based SFX synthesizer using WASM/Zig

Frenzy – Real-time prediction market where every trade resolves in 5 seconds

Structure and mechanistic basis of NrdR, a bacterial master regulator

The Kleene Star

A SECOND Sphinx detected in Egypt as scans hint at 'underground megastructure'

Anthropic's context-window.md is 18,501 tokens. 551 are content. I have notes

MCP Subagents

Amazon Dies: A Possible, Maybe Likely Future

Common Questions Asked in Investor Pitches