Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Arizona city sues over fire truck prices

Show HN: Monity.ai – Get notified when any website changes

Could End the RAM Crisis [video]

OpenAI's overhead will rise 20 percent for some workloads as it hardens security

Who's in charge? The fraught union of automation and human behavior

The Deadline Dividend: What Faster Inference Buys

How to tame AI's voracious appetite for energy

Cerebras CS-4 rack systems juice chips for every last drop of AI performance

Progress happens between practice sessions

Why crypto's best infrastructure companies stopped looking like crypto?

MAGA loyalists who collectively lost billions buying Trump stocks and crypto

New OllamaMQ v0.3.0

OpenAI lays out new security changes after its AI hacked Hugging Face

The Art of Prototyping

Obligatory one month without social media on my phone Substack essay

Whistleblower alleges political pressure,lack of evidence in antisemitism probes

Artificial intelligence boosts automated biolabs

Show HN: Interactive, animated architecture of any HuggingFace models

The Official History of Arm

Solo – a .so loader for static Linux binaries

Have an Agent Babysit Your Deployments

Mastodon 5.0: Laying the Foundation

Made a Lightweight Markdown Reader

Sci-Fi Author Jeffrey A Carver Passed Away

Minimal Zlang bytecode runtime booting in a bare-metal x86_64 kernel

The Mullet Stack: Syncing Types Between FastAPI/Pydantic and TypeScript

Soaring Diesel Prices Rip Across US Economy

Selective Applicative Functors

Running Ultrix 4.5 (2019)

The 400-Ton Linchpin of American Electrical Power