Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•8mo ago

Comments

nine_k•8mo ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•8mo ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•8mo ago

They mention this in the article.

spockz•8mo ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•8mo ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•8mo ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•8mo ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•8mo ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•8mo ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•8mo ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•8mo ago

This was very well written. Excellent article!

NetOpWibby•8mo ago

Is this like MessagePack for Haskell?

Maybe the database got it right

Jerome Powell Responds

The Living Internet

Internet History – One Page Summary

Additively Manufactured Dragonfly-Inspired Wings for Flapping Vehicles

Profession by Isaac Asimov (1957)

Defense Secretary tours SpaceX missile startup

Criminal charges against Fed Chairman for defying POTUS wishes

The U.S. Government Just Followed Through on Its Ban of DJI Drones

Tintcd: Directory-aware terminal theming for PowerShell. CD, but colorful

The Good Club

Show HN: 2k or Nothing – A Social Platform Dedicated to Long-Form Content

Jon Skeet Facts

OAuth 2.1 Dynamic Client Registration for AWS BedrockAgentCore Gateway

Serverless RAG and MCP on AWS with S3Vectors and Agentcore

Reverse-engineering another Ursa Major classic: the StarGate 323

Show HN: AeroTag – Tag-based workspace management for AeroSpace (macOS)

Hubble Telescope's Final Countdown: Could It Disappear Sooner Than Expected?

Token-Count-Based Batching: Faster, Cheaper Embedding Inference for Queries

Tuning Random Generators: Property-Based Testing as Probabilistic Programming [pdf]

Show HN: Built a course on buying small businesses – validating demand

A $400k payout is putting prediction markets in the spotlight

Matchbox Educable Noughts and Crosses Engine

Big Tech's Ugly Duckling: Can Snap Finally Execute?

Live Captions

You don't need a skill registry (for your CLI tools)

The US Empire is going supernova

Ogre 14.5 Released

Show HN: Instagram Saved Collection Downloader

Revolutionary eye injection saved my sight, says first ever patient