Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Adversarial Creation and Detection of AI-Generated Social Bot Content

Vantage – open-source CS2 player intelligence from Steam, FACEIT and Leetify

VictoriaMetrics: AI Policy

Show HN: FrontStep v0.5.2

Does Alpha School Work for Regular Kids?

GLM-5.3 – The Official Desktop Auditor for Z.AI's Cyber-Engine

Babel: A skyscraper where every Wikipedia edit becomes a brick

AI Software Development – What Does the Data Say?

PriEco: Customize Your Results Ranking

Traffic Modeling with Sumo: A Tutorial

DeepSeek V4 Flash at 278 tok/s, full precision, no quantization

Could a computer scientist build a brain?

The best way to sell a concentrated position at SpaceX

The Gestalt Principles

S.A.T.U.R.D.A.Y: serf-hosted speech to text AI assistant

Show HN: I built a spatial node canvas to fix LLM context drift

Show HN: Feature-packed 6MB Rust+Tauri free bookmarking app

1Password's new benchmark teaches AI agents how not to get scammed

Great music selection without overthinking

My open source project's donations no longer go through my bank account

Brazilian election filter in X For You timeline

AI assistance is not authorship

Simu5G: Simulator for 5G New Radio Networks

388 years ago, Galileo worked out why human giants can't exist

NASA processor is performing 500x faster than rad-hardened chips currently used

Do You Really Know Python? Quant Dev Mock Interview [video]

CIE 1931 XYZ colour space (2024)

Farside has been shut down

Brain turns listening inward during REM sleep, EEG recordings suggest

GitHub's push-farm spam hit 64% again – 106h live investigation