Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Show HN: I turned a 25 year old web platform into a HATEOAS multi-agent system

A competitor's ex-employee tried to sell me their secrets

Firefox extensions forced to declare misleading data transmission permissions

The Last Question

Can you proof read this post?

Ask HN: Is there a need for a new kind of antivirus or security application?

(Tech Report) DeepSeek-v4.1-Flash: Pushing Limits of KV Cache Compression [pdf]

I think I hate the internet

Hackers are stealing Claude tokens from subscribers

DeepSeek v4.1 Flash

Flowkon

AI Coding Assistants Almost Never Check Supply-Chain Trust Signals

PixoMonitor: Multi-location uptime and incident response platform

Astra for Coding: Why Are We Doing This Again?

Show HN: Doogi – Explore multiple AI responses and conversation paths

DeepSeek-v4.1-Exp

After outrunning Bolt, China's robot champion races towards real-world work

Framework cuts 32/64GB memory prices for 13 Pro, issues retroactive refunds

Building a readable, fast and safe language – looking for contributors

Scientists say August was Earth's hottest month they've measured

Governing GNOMEs: how the project's technical decision-making is evolving

Trump's FCC Threatens to Cancel Jimmy Kimmel Live over Interview with Democrat

Carbon Capture Won't Save Us Anytime Soon

Childcare: What the Science Says (2021)

The Rise of Retrofuturism

RL trained a 4B VLM to play GeoGuesser

ArchBench – Control Blender through chat

Shattered Pixel Dungeon v4.0.0

Hetzner's Cheap Cloud Tier Is Unavailable After Two Price Increases

You Don't Need to Write Separate Speeches for Men and Women