frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Show HN: Fast and Quality Code Chunking with Chonkie

1•snyy•1y ago
Hi HN,

We’re Chonkie (https://github.com/chonkie-inc/chonkie) — we build open source tools that help split documents into meaningful chunks for use with AI models.

When you use LLMs over large documents or codebases, you often need to break them into smaller parts to fit the model’s context window. Our chunkers do this in a smart way: they preserve structure and meaning, so only the most relevant pieces are passed into the model. This reduces hallucinations, avoids confusion, and improves performance and accuracy.

Today we’re launching our Code Chunker — a fast, structure-aware way to break down source code into high-quality, token-aware chunks.

How it works:

(See the code: https://github.com/chonkie-inc/chonkie/blob/main/src/chonkie...)

Code Chunker uses tree-sitter (https://tree-sitter.github.io/tree-sitter/) to parse your code into an abstract syntax tree (AST). It then recursively merges and groups nodes in a way that respects both code structure and token limits.

It supports all languages that tree-sitter supports, and is designed to preserve formatting and semantics. Large functions or class definitions won’t be split in the middle of a block — instead, we dive recursively into the AST to produce clean, coherent chunks that fit your configured token budget.

What it’s useful for:

  - Embedding-based code search

  - RAG (retrieval-augmented generation) over codebases

  - Long-context analysis of code

  - Preparing repos for fine-tuning or pretraining
Try it out:

  - Open source package: https://docs.chonkie.ai/chunkers/code-chunker

  - Hosted playground (free with account): https://cloud.chonkie.ai
Happy Chonking!

A Declaration of the Independence of Cyberspace (1996)

https://www.diplomacy.edu/resource/a-declaration-of-the-independence-of-cyberspace/
1•Bluestein•3m ago•0 comments

The Humans Who Made Grand Theft Auto

https://gamehistory.org/grand-theft-auto-1-development-team/
1•CharlesW•4m ago•0 comments

Wydatki – an expense tracker that is one HTML file and never goes online

https://wydatki-app.github.io/wydatki/#demo-en
1•adzak•5m ago•0 comments

Miami, the Real-Life City That Inspires Grand Theft Auto VI

https://www.nytimes.com/2026/11/18/arts/grand-theft-auto-vi-miami-florida.html
1•bookofjoe•5m ago•1 comments

GPT-6.2 Astra is the first model to beat Googlers' code golf records

https://bjin.github.io/go-codegolf-benchmark/#leaderboard
2•bjin•6m ago•1 comments

China's Spies Set Sights on A.I. Chips Years Before Nvidia Scandal

https://www.nytimes.com/2026/11/21/world/asia/china-nvidia-spy-chips-state-security.html
1•doener•6m ago•0 comments

The Alice Huang Associate Accused of Funneling Nvidia Chips to China

https://www.wsj.com/tech/li-ming-megaspeed-nvidia-chips-china-6260d836
1•doener•7m ago•0 comments

The Low Unemployment Streak Nears a Record

https://www.apollo.com/wealth/insights-news/insights/daily-spark/the-low-unemployment-streak-near...
1•akyuu•8m ago•0 comments

Half of Southern Europe's Young Adults Still Live with Their Parents

https://www.apollo.com/wealth/insights-news/insights/daily-spark/half-of-southern-europes-young-a...
3•akyuu•9m ago•0 comments

A Nation of Immigrants

https://www.witoldrybczynski.com/architects/a-nation-of-immigrants/
1•speckx•13m ago•0 comments

Don't Watch BBC, the GOV Don't Care You'll Still Pay from 2028 [video]

https://www.youtube.com/watch?v=JsBdd4PuTX4
1•dp-hackernews•13m ago•0 comments

AWS Lambda Adds GPU Instances for Event Functions on Lambda Managed Instances

https://aws.amazon.com/blogs/aws/aws-lambda-now-supports-gpu-instances-for-event-functions-on-lam...
1•ducviet00•18m ago•0 comments

In the Wilderness

https://rodrigoalvesvieira.com/2026/11/20/in-the-wilderness
1•ravieira•18m ago•0 comments

Butler Agents and Gardener Agents

https://nicholascharriere.com/blog/butlers-and-gardeners/
1•nichochar•19m ago•0 comments

Show HN: A Native GTK4/Libadwaita PDF Editor for Linux (AcroForms, PAdES, OCR)

https://github.com/word-sys/word-sys-pdf-editor
1•word-sys•20m ago•0 comments

Show HN: TripleClick – 220-line Mac menu bar app for trackpad middle click

https://github.com/wilu222/tripleclick
1•w222•20m ago•0 comments

How to Turn $4 into $351.0M

https://docshock.studio/research/follow-the-money/
2•docshock•22m ago•1 comments

Show HN: Podcast MCP

https://www.listennotes.com/api/mcp/
1•wenbin•22m ago•0 comments

Shield – Zero-dependency PII and secret masking for LLMs (MIT

https://github.com/PromptAudit-SaaS/shield
1•PromptSic•23m ago•0 comments

Changes on Freedesktop.org

https://blog.tenstral.net/2026/11/new-fdo-website.html
1•coryalthoff•23m ago•0 comments

Stop Committing Screenshot Noise

https://www.test2doc.com/blog/diff-detection/
1•speckx•24m ago•0 comments

William Shatner's Heavy Metal Album: WTF Is Heavy Metal?

https://wcsx.com/2026/10/20/william-shatners-heavy-metal-album/
1•Michelangelo11•25m ago•0 comments

Show HN: I built a risk map of the Army's planned lead azide detonation in Iowa

https://getlucera.com/iowa-lead-detonation
1•slawton3•26m ago•0 comments

Ask HN: NixOS, why no init freedom?

1•br0ceph•28m ago•1 comments

Practical cryptanalysis with MiniMax's latest Internal Model

https://www.minimax.io/blog/practical-cryptanalysis-with-minimaxs-latest-internal-model
2•dnw•28m ago•0 comments

Value Creation vs. Job Preservation

https://chrlschn.dev/blog/2026/11/value-creation-vs-job-preservation/
1•CharlieDigital•32m ago•0 comments

Silicon Valley Is Suffering from Model Collapse

https://www.reddit.com/r/technology/s/pd5TGWsYDf
3•sylvainkalache•35m ago•3 comments

The vibe-coding revolution is coming to all the gadgets

https://www.theverge.com/column/1039139/vibe-coding-gadgets-tv-camera
2•dkobia•35m ago•0 comments

Isaac Asimov: Jokester

https://lecturia.org/en/short-stories/isaac-asimov-jokester/29130/
2•thunderbong•35m ago•0 comments

But what is the Riemann hypothesis? Visualizing analytic continuation [video]

https://www.youtube.com/watch?v=sD0NjbwqlYw
1•binyu•36m ago•0 comments