Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Space X and Tesla exploring merger, CEO Elon Musk says

In Hilbert space, all things are quantumly possible

Video Game Reverse Engineering Community Shuts Down

Merging SpaceX and Tesla would be a "good idea," Musk says

OpenAI CLI – Official CLI for the OpenAI API

Is Complexity an Illusion? (2024)

OpenAI – 12 hour cache (API) available and "cache forever" for work/codex users

Show HN: Litedeck – Agentless server management GUI over plain SSH

Show HN: Ida, the Mac assistant Siri should have been

NCM: Parallel Coarse-to-Fine Text Generation

I wish you didn't want to see Nathan Fielder's Elizabeth Holmes documentary

T3 Code now synced between GitHub and GitCafe

The Future of CSS

Removing the Leap Second

GitHub package firewall

Indian CA's private keys compromised – one of them was the number '4'

Running Agent Factory for Months

Good type against all odds

Learning to live with Waymos

Guarantee Ukraine's Ballot, Not Just Its Border

SimCity 2000, in the style of OpenTTD

Show HN: Unterm – A pure-Rust terminal for humans and AI agents

Finland to Build a Million Landmines as Russian Threat Looms

Show HN: Gemma 4 26B at 100 tok/s on a Mac, using the Neural Engine and 2GB RAM

The Gospel According to Peter (Thiel)

Cveelo – Turn your CV into a portfolio site, with hand-made templates

Agentitter – a tiny plain-text social network you use through your AI agent

Dataville Takes the Guesswork Out of Data for AI

Evolution of Gas Sensors in the Mining Industry

Ask HN: How have LLMs changed freelance dev?