Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

How to be useful as a software architect

Elevated Errors for Opus 5

I built a jav data API, need honest feedback

Show HN: OpenStrap – use a WHOOP 4.0 band without a subscription

Show HN: Tawrida – Global Trade Tools – Export-Document Generators

Anthropic should learn from those cotton-picking socialists

Practice your YC interview against Garry Tan's gstack specialists, on a meeting [video]

High Volume Transaction Processing Without Concurrency Control (1997)

Ruff v0.16.0 – Significant new updates – 413 default rules up from 59

Build a self-scaling OCR pipeline with Qwen 3.5 and Kubernetes

Claude Code has a hardcoded instruction telling Opus 5 not to use subagents

The most capable resume editor I could build, as a solo dev

The diminishing returns of productivity culture (2021)

Show HN: Reproducibility Benchmark a Risk Quantitative Model

What will more intelligence do for us?

HomeLab #2: Immutable GitOps homelab and MikroTik hardening

If you feel the need to tell me your answer is AI-based, work it more

The Context Bootloader

House AI 'kill switch' bill unveiled as OpenAI hack raises alarms

Modeling Facts and Reactions with Domain Events

Anthropic secures its AI-native software development lifecycle

Visaism (Part 1): India as a Waiting Room

Fallback Font

Yann LeCun's Bet That Intelligence Starts in the World

Teach Yourself Programming in Ten Years (1998)

YouTube Comments in .csv

A protocol for AI agent workspace state and effect management

Reth ExEx spawn_critical_task terminates node on any plugin panic

Claude Code Cut Their System Prompt by 80%. Does That Work for Small Models Too?

Janet Lustgarten: An Oral History of the Founding of Kx Systems