Packed Data Support in Haskell

https://arthi-chaud.github.io/posts/packed/

77•matt_d•1y ago

Comments

nine_k•1y ago

> Introducing the ‘packed’ data format, a binary format that allows using data as it is, without the need for a deserialisation step. A notable perk of this format is that traversals on packed trees is proven to be faster than on ‘unpacked’ trees: as the fields of data structures are inlines, there are no pointer jumps, thus making the most of the L1 cache.

That is, a "memory dump -> zero-copy memory read" of a subgraph of Haskell objects, allowing to pass such trees / subgraphs directly over a network. Slightly reminiscent of Cap'n Proto.

90s_dev•1y ago

We are always reinventing wheels. If we didn't, they'd all still be made of wood.

Zolomon•1y ago

They mention this in the article.

spockz•1y ago

It reminds me more of flat buffers though. Does protobuf also have zero allocation (beyond initial ingestion) and no pointer jumps?

cstrahan•1y ago

No, one example of why being variable sized integers.

See https://protobuf.dev/programming-guides/encoding/

carterschonwald•1y ago

One thing that sometimes gets tricky in these things is handling Sub term sharing. I wonder how they implemented it.

tlb•1y ago

> the serialised version of the data is usually bigger than its in-memory representation

I don’t think this is common. Perhaps for arrays of floats serialized as JSON or something. But I can’t think of a case where binary serialization is bigger. Data types like maps are necessarily larger in memory to support fast lookup and mutability.

nine_k•1y ago

I suppose all self-describing formats, like protobuf, or thrift or, well, JSON are bigger than the efficient machine representation, because they carry the schema in every message, one way or another.

IsTom•1y ago

If you use a lot of sharing in immutable data it can grow a lot when serializing. A simple pathological example would be a tree that has all left subtrees same as the right ones. It takes O(height) space in memory, but O(2^height) when serialized.

gitroom•1y ago

honestly i wish more stuff worked this way - fewer hops in memory always makes me happy

lordleft•1y ago

This was very well written. Excellent article!

NetOpWibby•1y ago

Is this like MessagePack for Haskell?

Show HN: Saar Agentic Orchestration Platform

OCaml 5.5.0 Released

Hilarious German Compound Nouns you won't want to miss

Reality has a surprising amount of detail (2017)

Show HN: Onbalance – privacy-first cashflow planning app got a multiuser update

GitHub Copilot improves productivity 40%

Data centers become the face of AI backlash

Productivity: Bookmarkr Chrome plugin with Visual and organized grid

Bitcoin is stuck near $64,000 as ETF outflows reach a sixth week

Black Box Probing: A Security Analysis of Xiaomi's MJA1 Secure Chip

Show HN: Prismag – Per-block model routing for the terminal and any IDE

UUID: NewV7() always generates a UUID with 7000 on browsers (Golang)

QUIC is more than a replacement for TCP

In Mizoram, India, the Shops Have No Shopkeepers (2020)

Turning spoken commands into JSON tool calls on iPhones

Lessons from Building Evals for Financial AI Agents

I moved a broker's virtualization from Hyper-V to Proxmox with zero downtime

Callback 8020 by Commodore – The Flip Phone Between Dumb and Smart

Minia2a – A Marketplace Where AI Agents Earn Money

Databricks vs. AWS managed service which one fits your need?

Natural Selection Maximizes Fisher Information

Safebucket – an open source file sharing platform with pluggable infrastructure

Yann LeCun „World Models: Enabling the Next AI Revolution" [video]

Optimizing [sqlx:test] rebuild time

Networking tool to host WireGuard servers with only outbound connections

Anthropic's Mythos mess just keeps getting more complicated

Agile and Coding: An Agent- and Human-Friendly Architecture

PivCo-Huffman "Merge" Operations

Show HN: Cyclearchive.com – searchable archive of historical cycling literature

Interlude: Using the Index Registers Effectively on the Z80