Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•12mo ago

Comments

nine_k•12mo ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•12mo ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•12mo ago

They mention this in the article.

spockz•12mo ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•12mo ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•12mo ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•12mo ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•12mo ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•12mo ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•12mo ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•12mo ago

This was very well written. Excellent article!

NetOpWibby•12mo ago

Is this like MessagePack for Haskell?

"Mathematics is a fundamentally *human* story"

Sandbox filesystem and network access without requiring a container

ByteCode C2 is now open source. A C2 framework that bypasses Defender

How Tech Loses Out over at Companies, Countries and Continents

Preliminary in-progress RISC-V "P" Extension

The model is still not the product

Peter Thiel Flees to Argentina

Show HN: Phonetic Formatter – offline English text to IPA on iPhone and iPad

Show HN: Economic growth is a power law

Why C Remains the Gold Standard for Cryptographic Software

40 Years Ago, a Nuclear Catastrophe at Chernobyl

Codex MSN Interface

Headless websites and the cost of engineering vanity

Quick tutorial to get a blog online from Org Mode thanks to Org Social

APL is more French than English

The Knight Programming Language

Exposing Floating Point – Bartosz Ciechanowski

Seven database engines in a single Rust binary

Tip: Web requests should not be measured in Hz [Hertz]

Self-Updating Screenshots

Open grid data has a public benefit

Airprompt – SSH into your Mac from your phone for AI agent prompts

Show HN: A community powered global network of probes

The Scrum-to-POM Transition Is a Role Repositioning Event

Pytest-cloudreport – local HTML reports and flaky-test detection for pytest

Blueprint: AI Hardware Design

US is making Europe pay dearly for its half-hearted electrification

The reporters at this news site are AI bots. OpenAI's super PAC is funding it

San Francisco must preserve the birthplace of the Mission burrito

Enterprises Are Rethinking Kubernetes