Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Git Hash Chain Malleability

Neural Geometry in Vision Models with Block-Sparse Featurizers

J-Space: Where Claude silently performs reasoning steps

Bond Traders Stunned as Losses on SpaceX's New Debt Keep Growing

APS-CYBER: A state-transition security platform to detect attacks before impact

FrankenMarkdown

Bees 'facial expressions' may be a sign of their inner lives

Why the rise of open source AI isn't hurting Anthropic yet

Silent speech with ultrasound

Who's Keeping Fossil Fuels Alive? Taxpayers

MIL – Moments in Life

Composing TLA+ Specifications with State Machines

Constructing a lower-bound estimate of the global number of insect species

The Wisdom of Holden Caulfield

Create Your Language

Your Idea Is Brilliant, Your Idea Is Worthless (KS Lesson #204) (2016)

U7in-32tb.224xlarge

Disney accuses US media regulator of trying 'to sit in the editor's chair'

Qualcomm acquires Nexa AI, open-sources GenAI runtime for Hexagon NPUs

Chorus: A fast WAL for object storage

Google Earth Pro Desktop downloads will be unavailable starting June 2027

EU AI Act becomes applicable Aug 2: an engineering checklist

Agent-CI: Run GitHub Actions on Your Machine

Why I don't have a girlfriend: An application of the Drake Equation to love [pdf]

Graphene can hold multiple states of superconductivity, a new study finds

Don't build your castle in other people's kingdoms (2021)

Airbus to make first foray into engine manufacturing with hydrogen tie-up

Own Your Weights

The Science Behind Why Soccer Players the 26 World Cup Are Cutting Their Socks

Show HN: Convergo – plan/build review loops for coding agents