frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Show HN: Fast and Quality Code Chunking with Chonkie

1•snyy•1y ago
Hi HN,

We’re Chonkie (https://github.com/chonkie-inc/chonkie) — we build open source tools that help split documents into meaningful chunks for use with AI models.

When you use LLMs over large documents or codebases, you often need to break them into smaller parts to fit the model’s context window. Our chunkers do this in a smart way: they preserve structure and meaning, so only the most relevant pieces are passed into the model. This reduces hallucinations, avoids confusion, and improves performance and accuracy.

Today we’re launching our Code Chunker — a fast, structure-aware way to break down source code into high-quality, token-aware chunks.

How it works:

(See the code: https://github.com/chonkie-inc/chonkie/blob/main/src/chonkie...)

Code Chunker uses tree-sitter (https://tree-sitter.github.io/tree-sitter/) to parse your code into an abstract syntax tree (AST). It then recursively merges and groups nodes in a way that respects both code structure and token limits.

It supports all languages that tree-sitter supports, and is designed to preserve formatting and semantics. Large functions or class definitions won’t be split in the middle of a block — instead, we dive recursively into the AST to produce clean, coherent chunks that fit your configured token budget.

What it’s useful for:

  - Embedding-based code search

  - RAG (retrieval-augmented generation) over codebases

  - Long-context analysis of code

  - Preparing repos for fine-tuning or pretraining
Try it out:

  - Open source package: https://docs.chonkie.ai/chunkers/code-chunker

  - Hosted playground (free with account): https://cloud.chonkie.ai
Happy Chonking!

Anthropic's IPO Bust Deepens $4T AI Wipeout on Wall Street

https://www.wsj.com/finance/stocks/anthropics-ipo-bust-punctuates-4-trillion-ai-wipeout-on-wall-s...
1•qlte•1m ago•0 comments

Show HN: Made a career portal for EU agencies

https://civicjobs.eu/
1•usernametaken29•3m ago•0 comments

Ask HN: How did Hacker News get its name?

1•illidan1000•3m ago•0 comments

What it takes to be a super pooper

https://www.bbc.com/future/article/20261117-faecal-transplants-what-it-takes-to-be-a-super-pooper
1•AareyBaba•4m ago•0 comments

Pasilalinic-Sympathetic Compass: The Snail Telegraph

https://www.amusingplanet.com/2026/11/pasilalinic-sympathetic-compass-snail.html
1•bookofjoe•4m ago•0 comments

The Riemann Hypothesis Falls (reaction) [video]

https://www.youtube.com/watch?v=sMZMpsHeFKc
1•nomilk•4m ago•0 comments

Kokoro Read-Aloud – select text in any app, hear it read locally

https://github.com/sayed-qutob-work/kokoro-read-aloud
1•sayedqutobwork•4m ago•0 comments

The Tool, the Dream, and the Privacy of Thought

https://medium.com/@smoking2man/the-tool-the-dream-and-the-privacy-of-thought-on-the-displacement...
1•Aldus551•6m ago•0 comments

I brought Omarchy to an iPad (and what it took to get there)

https://twitter.com/henrikklee/status/2114409814445195551
2•raybb•8m ago•1 comments

A Procedural File Format Could Be Better Suited for Navigation Software

https://bitilia.com/blog/procedural-file-format-navigation-software/
1•bitilia•8m ago•0 comments

Show HN: A mechanical computer that runs the Rule 110 cellular automata [video]

https://www.youtube.com/shorts/iIKQuGVjefM
1•tim_hutton•9m ago•1 comments

Working as Intended [video]

https://www.youtube.com/watch?v=Z6_ENp_fO_Y
1•largbae•10m ago•0 comments

TensorLean: Formal verification framework for GPU kernels at the PTX level

https://tensorlean.ai/
1•matt_d•10m ago•0 comments

OPM Data Was Breached on the Same Day as the FAA Ground Stop, per Internal Email

https://www.muellershewrote.com/p/breaking-opm-data-was-breached-on
2•MadrasThorn•12m ago•0 comments

Trump Says He Will Penalize Companies That Build Data Centers Outside the U.S.

https://www.wsj.com/politics/policy/trump-to-hold-white-house-press-conference-following-trip-to-...
2•doener•15m ago•0 comments

AI Trade Fears, Anthropic IPO Aftermath Prompts Tech Selloff

https://www.bloomberg.com/news/live-blog/2026-11-20/markets-turmoil
1•blondie9x•15m ago•1 comments

Build 3D models with Muse Code

https://dev.meta.ai/resources/blog/build-3d-models-with-muse-code
7•nimmerland•15m ago•0 comments

Pill to treat pancreatic cancer hailed as 'biggest breakthrough in decades'

https://www.theguardian.com/society/2026/nov/20/pill-pancreatic-cancer-breakthrough-daraxonrasib
1•rafaelc•16m ago•0 comments

Marcin Borkowski: Copying the current location improved again

https://mbork.pl/2026-11-16_Copying_the_current_location_improved_again
4•millspore•21m ago•0 comments

Consumers have grown accustomed to deceptive design online

https://www.uwasa.fi/en/newshub/news/hidden-cancellation-buttons-and-fake-urgency-consumers-have-...
2•geox•22m ago•0 comments

AI is building me a web browser

https://blog.senko.net/ai-is-building-me-a-web-browser
4•taubek•22m ago•0 comments

An update on Blueprints and the path to UE6

https://www.unrealengine.com/news/an-update-on-blueprints-and-the-path-to-ue6
1•boriskourt•22m ago•0 comments

GPU Benchmarks Hierarchy 2026

https://www.tomshardware.com/reviews/gpu-hierarchy,4388.html
1•jonbaer•22m ago•0 comments

Brake Check: Hijacking Trailer Brakes over Bluetooth

https://elementdigital.com.au/blog/brake-check-trailer-brakes-over-bluetooth/
2•DeepLogin•22m ago•0 comments

Show HN: SocialGTM – Find your first customers in conversations on social

https://socialgtm.app
1•sayandedotcom•23m ago•0 comments

Actual Curl Rewrites in Rust

https://daniel.haxx.se/blog/2026/11/17/actual-curl-rewrites-in-rust/comment-page-1/#comments
1•todsacerdoti•24m ago•0 comments

GTA VI Shatters Box Office Records with $3.79B in Day One Retail Sales

https://www.businesswire.com/news/home/20261120446139/en/Grand-Theft-Auto-VI-Shatters-Entertainme...
3•haunter•26m ago•0 comments

LegisComply – Mask sensitive details in AI prompts

https://www.legiscomply.com/?sample=proposal-review
1•xantin•27m ago•0 comments

Is Wonder the dinner company for our anti-social times?

https://www.cnn.com/2026/11/20/business/wonder-food-delivery-marc-lore
1•mooreds•28m ago•0 comments

Why are all the indie-friendly products dying at the same time?

https://cassidoo.co/post/dying-indies/
2•mooreds•28m ago•0 comments