Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

CueHQ finds the threads asking for what you offer and drafts a reply

OpenAI might rebrand its data centers as 'AI compute campuses'

Show HN: A free perfume MCP server and API, no key (benchmark inside)

Proofs are cheap now. Specs are next

Amazon Kiro proves properties of its own harness in Lean

I converted 661M rows of Texas oil and gas data from mainframe dumps

Show HN: Scan-to-vector – rebuild scanned pages as vector PDFs with an AI agent

Typographical Mistakes That Had Major Historical Consequences

Show HN: Chinese sentence checker with pinyin and English explanations

Musk Faces Setback as SpaceX and Tesla Stocks Tumble on Midterm Election Results

Axion 1.0, an open agent foundation model

An AI agent's first 73 days, counted: 0 buyers, 8 pieces, 2 readers

Automated Secret Rotation Is Chaos Engineering for Your Identity Plane

The Post-Code Generation

Swatch Internet Time: 1k Beats per Day

Communist Hungary and its Arab informers

3D-printed brick phone brings back the art of simply answering calls

Show HN: The HTML Library of Babel

Snowflake's AI_PARSE_DOCUMENT got an agentic mode

I Wrote a Book: The Art and Science of LLM Jailbreaking

SpaceX Selects Qualcomm to Accelerate Starlink Mobile's Nationwide 5G Network

Show HN: I wanted OpenSCAD with fillets and a C-like language. So I made one

App that feels better then pintrest in wallpapers

Eric Conveys an Emotion

How we built Lovable, and why we keep rebuilding it

Old, Polluting Jet Engines Are Being Scavenged to Power Data Centers

Mocktastic: A Meshtastic Node on Arduino

JobsWorkers – A marketplace for home and personal services

MCP vs. API vs. CLI: How should AI agents connect to your app?

Developers Are Getting RAIded