frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Show HN: Fast and Quality Code Chunking with Chonkie

1•snyy•1y ago
Hi HN,

We’re Chonkie (https://github.com/chonkie-inc/chonkie) — we build open source tools that help split documents into meaningful chunks for use with AI models.

When you use LLMs over large documents or codebases, you often need to break them into smaller parts to fit the model’s context window. Our chunkers do this in a smart way: they preserve structure and meaning, so only the most relevant pieces are passed into the model. This reduces hallucinations, avoids confusion, and improves performance and accuracy.

Today we’re launching our Code Chunker — a fast, structure-aware way to break down source code into high-quality, token-aware chunks.

How it works:

(See the code: https://github.com/chonkie-inc/chonkie/blob/main/src/chonkie...)

Code Chunker uses tree-sitter (https://tree-sitter.github.io/tree-sitter/) to parse your code into an abstract syntax tree (AST). It then recursively merges and groups nodes in a way that respects both code structure and token limits.

It supports all languages that tree-sitter supports, and is designed to preserve formatting and semantics. Large functions or class definitions won’t be split in the middle of a block — instead, we dive recursively into the AST to produce clean, coherent chunks that fit your configured token budget.

What it’s useful for:

  - Embedding-based code search

  - RAG (retrieval-augmented generation) over codebases

  - Long-context analysis of code

  - Preparing repos for fine-tuning or pretraining
Try it out:

  - Open source package: https://docs.chonkie.ai/chunkers/code-chunker

  - Hosted playground (free with account): https://cloud.chonkie.ai
Happy Chonking!

Code was my second mother tongue. Now everyone speaks it. What could go wrong?

https://blog.ngxson.com/going-deaf-in-my-own-language
1•ngxson•49s ago•0 comments

Plv8: V8 Engine JavaScript Procedural Language Add-On for PostgreSQL

https://github.com/plv8/plv8
1•thunderbong•4m ago•1 comments

Show HN: Git blame for AI code, down to the agent session

https://keepplain.com/blame
1•keepplain•6m ago•0 comments

Cappz – AI agents sell work to each other, paid per stage on XRPL

https://github.com/CAPPZ-ai/cappz-sdk
1•fastflair•8m ago•0 comments

Trump vows to work with Democrats to cap credit card interest rates at 10%

https://apnews.com/article/trump-credit-cards-interest-rates-warren-banks-098a69b520e520e535eebe5...
1•petethomas•13m ago•1 comments

Macaron – Turn customer messages into fixes ready for review

https://macaron.sh
1•garagemanda•14m ago•0 comments

Americas Mexico detains top cartel boss 'El Chapo Isidro,' setting off violence

https://www.npr.org/2026/11/21/nx-s1-6049672/cartel-boss-el-chapo-isidro-detained
1•blinding-streak•15m ago•0 comments

Bayer reaches Roundup settlement that could cost the company billions

https://www.npr.org/2026/11/21/nx-s1-6049684/roundup-bayer-hodgkins-settlement
1•blinding-streak•17m ago•0 comments

Measuring nanometers by looking at your reflection [video]

https://www.youtube.com/watch?v=k4AEZ35VklM
1•ivanjermakov•17m ago•0 comments

Field Guide: Running Qwen3.8 Locally

https://igorstechnoclub.com/field-guide-running-qwen38-locally/
1•Igor_Wiwi•18m ago•0 comments

Kvdiff – Local proxy showing the byte breaking your LLM prompt cache

https://github.com/flipcyde101/kvdiff
1•flipcyde101•19m ago•0 comments

Show HN: AGI: Robots Awake – a browser maze shooter

https://marcvscouto.itch.io/agi-robots-awake
2•mvbrasil•25m ago•1 comments

Zlin City: Diorama – Closed Playtest Announcement [video]

https://www.youtube.com/watch?v=5T6dq8adEvo
1•phil-pickering•26m ago•1 comments

Internet Archive employees seek to form a union

https://www.heise.de/en/news/Internet-Archive-employees-seek-to-form-a-union-11533621.html
1•Tomte•29m ago•0 comments

Making models predictable makes their errors more random

https://www.youtube.com/watch?v=-MiM5xfAe2c
1•gitowiec•29m ago•0 comments

Anthropic Opened Its Books. What Is It Worth? [video]

https://www.youtube.com/watch?v=Ic1Jf7VlXRM
1•Bluestein•29m ago•1 comments

Fleet – manage multiple harnesses with a central agent in a TUI

https://github.com/sturlabragason/fleet
2•sturla-bragason•32m ago•1 comments

SVG loading (rendering, conversion) in Castle Game Engine thanks to FPC 3.3.1

https://castle-engine.io/wp/2026/11/21/svg-support/
1•mariuz•34m ago•0 comments

Show HN: Lightweight Digital Piggybank

https://app.supademo.com/video/cmwqynzsi0efcqm24gcrc2tl7
1•hito20•37m ago•0 comments

Alice Huang's links to the chip underworld

https://www.ft.com/content/f3ad8a81-3890-4a8a-8853-ff0a94b08ba9
1•Betelbuddy•41m ago•0 comments

A Better Vocabulary for Testing

https://alperenkeles.com/posts/vocab-for-testing/
1•alpaylan•42m ago•0 comments

Noirvpn – a 10x faster openvpn in Odin

https://github.com/cosiniumdev/noirvpn
2•ldkjhg•43m ago•1 comments

Cursor may stop using Chinese open-weights AI models

https://casrai.org/news/house-committees-cursor-chinese-open-weight-ai-models-report-2026
2•ComputedCamp995•43m ago•1 comments

Kneel Before the Emperor of Mecatol Rex (2024)

https://ludic.mataroa.blog/blog/kneel-before-the-emperor-of-mecatol-rex/
2•Tomte•43m ago•0 comments

Show HN: Can AI Rewrite OpenCode in 90% Rust?

https://www.youtube.com/watch?v=SwX4OoxMaKM
2•latent-9•44m ago•0 comments

PcapAnalyzer – self-hosted pcap analyzer with 27 problem detectors

https://github.com/sebaandrada-cell/PcapAnalyzer
9•sebaandrada•48m ago•1 comments

The Last Great Revolution

https://www.lesswrong.com/posts/ZxBaCDjxyxKaLHqGf/the-last-great-revolution
1•paulpauper•48m ago•0 comments

To code well, avoid blunders

https://www.seangoedecke.com/to-code-well-avoid-blunders/
4•gfysfm•48m ago•1 comments

Tesla Reality View Live Visualization Update

https://twitter.com/Tesla/status/2124222447855743019
3•hnburnsy•48m ago•0 comments

Things About the Insanity of Sand

https://kenorland.substack.com/p/the-insanity-of-sand
1•paulpauper•49m ago•0 comments