Show HN: litelink - local-first, embedded stream capture into Iceberg tables

Hi HN! I just wanted to share litelink a local-first, embedded capture library I built in python (code is heavily AI generated but designed and reviewed by yours truly). Litelink supports a single writer per stream.

I've been doing a lot of development and deployments on tiny VMs (2 vCPU, 8GB, 50-100GB disk) and didn't want the complexity or cost of managing central brokers (Kafka), databases (Postgres), and CDC/connectors just to get queryable WebSocket stream capture running.

With litelink, you configure a log in code, and end-to-end setup takes <5 minutes (see the example scripts in the repo). The log is itself an Iceberg table (actually two: a local and archive table), so there's no second copy of your data to keep in sync or connector to manage.

I'm sure there are still bugs, but I recently migrated all the capture feeds for a personal research project to litelink, and the experience has been night and day. Before that, I'd hand-rolled a capture system and was dealing with all the issues you'd expect:

- The small file problem (the worst single stream contained 200,360 files averaging 46 KB some files literally containing zero rows)

- Managing compaction and retention

- Managing a custom hive-partitioned layout on disk and in S3

- In-memory buffering that cost data on SIGKILL

Ingesting data is easy(ish). Maintaining it is not. That's exactly why I wrote litelink (so you don't have to think about maintenance). I'll post some before/after stats in a comment below. Also, the hand-rolled capture wasn't actually durable (due to the in-memory buffering) and litelink capture is. You can read about the design and exactly how it works in the docs in the repo. Feel free to ask any questions here though :)

I tried to channel the same ethos as LanceDB/Iceberg/SQLite. Everything runs local first without a network connection required. I've tried to abstract the complexity of stream/data lifecycle maintenance away behind a few public library methods. Hopefully someone else finds this useful! Let me know what you think.

repo: https://github.com/nhobin219/litelink

spec: https://github.com/nhobin219/litelink/blob/main/docs/SPEC.md

pypi: `pip install litelink`

Show HN: A website anyone can edit through auto-merged GitHub PRs

New York Times and The Athletic workers demand company scrap Kalshi deal

A Transformer Plays Tic-Tac-Toe

Justifying text by hand, now at 35 columns

RSA-260 Factored

Astronomers Detect a 10-Sided Structure in Saturn's Atmosphere

Fuck Off as a Service (Foaas)

Show HN: A newsletter about systems performance, scalability, and efficiency

AI could have written this post

Sitelo – Static Site Generator

Coloring the Snub Dodecahedron

Active Inference: The Free Energy Principle in Mind, Brain, and Behavior

Show HN: DumbDetector – Community signals for AI tool problems

ARKM KERNEL approaching you soon

Ask HN: Is SpaceXAI taking Claude down too?

Balrogg: Demonically compacting (up to 15%) lossless Vorbis/Opus recompressor

The Cognitive Revolution (When Machines Do the Thinking)

ICE Has a $2M Contract for Spyware That Can Hack Phones Without a Click

Show HN: Litelink – local-first, embedded stream capture into Iceberg tables

Raycast now requires a subscription for BYOK and local models

Show HN: Pangolin – SSO and WireGuard instead of API keys for LLM access

Life Is Bigger Than GPUs

Paid Phones

Vibe Coding Does Not Have One Definition, It Has One per Developer

An AI agent sliced through an enterprise defenses in 10 hours

21st Revision of Patches for Getting Linux to Shutdown Devices Asynchronously

Live Atomic Spatial Audio

Show HN: Eliminating low-LTV support ticket spam with grounded RAG

Elevated Errors for Multiple Models

GLP-1 treatment late in life extends lifespan in animal model

Show HN: Git-chain – Simplify branch dependency management in Git

Comments