frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Show HN: Fast and Quality Code Chunking with Chonkie

1•snyy•1y ago
Hi HN,

We’re Chonkie (https://github.com/chonkie-inc/chonkie) — we build open source tools that help split documents into meaningful chunks for use with AI models.

When you use LLMs over large documents or codebases, you often need to break them into smaller parts to fit the model’s context window. Our chunkers do this in a smart way: they preserve structure and meaning, so only the most relevant pieces are passed into the model. This reduces hallucinations, avoids confusion, and improves performance and accuracy.

Today we’re launching our Code Chunker — a fast, structure-aware way to break down source code into high-quality, token-aware chunks.

How it works:

(See the code: https://github.com/chonkie-inc/chonkie/blob/main/src/chonkie...)

Code Chunker uses tree-sitter (https://tree-sitter.github.io/tree-sitter/) to parse your code into an abstract syntax tree (AST). It then recursively merges and groups nodes in a way that respects both code structure and token limits.

It supports all languages that tree-sitter supports, and is designed to preserve formatting and semantics. Large functions or class definitions won’t be split in the middle of a block — instead, we dive recursively into the AST to produce clean, coherent chunks that fit your configured token budget.

What it’s useful for:

  - Embedding-based code search

  - RAG (retrieval-augmented generation) over codebases

  - Long-context analysis of code

  - Preparing repos for fine-tuning or pretraining
Try it out:

  - Open source package: https://docs.chonkie.ai/chunkers/code-chunker

  - Hosted playground (free with account): https://cloud.chonkie.ai
Happy Chonking!

Style Is Consistent Constraint

https://stephango.com/style
1•tosh•1m ago•0 comments

Laptop Can Hear You Without a Microphone. Proof

https://twitter.com/whyvanamali/status/2122276417031585879
1•mrkn1•1m ago•0 comments

Show HN: Own your AI at full speed

https://www.rightnowai.co/
1•OsamaJaber•2m ago•0 comments

Challenging Consensus, Scientist Says Photos Show Epstein Was Killed

https://www.nytimes.com/2026/11/16/us/jeffrey-epstein-autopsy-michael-freeman.html
1•ParentiSoundSys•2m ago•0 comments

A Generation Revolts

https://www.noemamag.com/a-generation-revolts/
1•littlexsparkee•3m ago•0 comments

Huang's Ouster Fuels Doubts About Nvidia's Support for Neoclouds

https://www.bloomberg.com/news/articles/2026-11-16/huang-s-ouster-fuels-doubts-about-nvidia-s-sup...
1•petethomas•4m ago•0 comments

Baptize.Link – clean tracking cruft out of links with paste or url append

https://baptize.link/
1•RobLach•4m ago•1 comments

Perplexity Monitor API

https://www.perplexity.ai/hub/blog/introducing-perplexitys-monitor-api
1•denysvitali•4m ago•0 comments

Sex in Public

https://lmnt.me/blog/sex-in-public.html
1•speckx•6m ago•0 comments

What Happens to Work After a Timeout?

https://octri.dev/blog/what-happens-to-work-after-a-timeout
1•khalidkhair•6m ago•0 comments

Paul Christiano on the Dwarkesh Podcast [video]

https://www.dwarkesh.com/p/paul-christiano-2
2•keito•6m ago•0 comments

Show HN: Sample – Music Player

https://sample-music-player.netlify.app/
1•sidd_sarkar•7m ago•0 comments

Apple adds 'face unlock' to the iPhone Duo, but it's not Face ID

https://techcrunch.com/2026/11/16/apple-adds-face-unlock-to-the-iphone-duo-but-its-not-face-id/
1•lavendermon•9m ago•0 comments

The Rise and Fall of Heathkit – Part 1: Early Days

https://www.eejournal.com/article/the-rise-and-fall-of-heathkit-part-1-early-days/
1•CharlesW•9m ago•0 comments

Show HN: Self-Hosted Contact Forms for Your Web App in Go

https://github.com/VictorioBerra/miform
1•Lord_Zero•10m ago•0 comments

GenAI Is Erasing the Mid-Level Engineer Promo Signal

https://swaranga.dev/posts/who-are-we-actually-promoting/
1•swaranga•12m ago•0 comments

Sopro V2: private, fast, on-device text-to-speech

https://research.haloneuro.ai/posts/sopro-v2
1•gmays•13m ago•0 comments

Policitian Marco Rubio rewrote Intel findings behind Jensen Huang's ouster

https://twitter.com/coinbureau/status/2121622935811051710
4•bigwheels•13m ago•1 comments

AI Agents, Why Are You All Saying the Same Thing?

https://www.zmistandcopy.com/blog/ai-agents-why-are-you-all-saying-the-same-thing
1•kabrosimova•13m ago•0 comments

Evolving Infra: Agent-Adaptive LLM Serving at Scale

https://arxiv.org/abs/2611.12304
1•matt_d•14m ago•0 comments

Can you write the code to run an airport control tower?

https://holdingpattern.dev/
2•shawnzam•14m ago•1 comments

Streamhouse: A shared data architecture for the age of AI

https://www.streamhouse.com/
1•saikatsg•15m ago•0 comments

From Cybersecurity Platforms to Systems

https://strategyofsecurity.com/p/from-cybersecurity-platforms-to-systems
1•mooreds•16m ago•0 comments

Quebecor makes $31B offer to acquire Telus

https://www.cbc.ca/news/business/quebecor-telus-acquisition-offer-9.7423299
1•ejj28•17m ago•0 comments

Nvidia Announces Collaboration with IBM and Fermi National Accelerator Lab

https://nvidianews.nvidia.com/news/nvidia-announces-collaboration-with-ibm-and-fermi-national-acc...
4•saikatsg•18m ago•1 comments

PowerShell 7.7 is released – as an MSIX

https://oofhours.com/2026/11/15/powershell-7-7-is-released-as-an-msix/
3•speckx•20m ago•0 comments

Does weight-space programming need to touch the hidden layers?

https://probe-graft.vercel.app/2/
3•vaibhavarduino•21m ago•0 comments

Show HN: Solipsist – Work with Pi agents using Markdown

https://solipsist.sh/
2•pulkitsh1234•22m ago•0 comments

One malfunctioning router grounded every flight in America last week

https://www.techspot.com/news/114567-how-one-malfunctioning-router-arkansas-grounded-every-flight...
3•alok-g•24m ago•0 comments

Laid off or leaving, start with the list nobody writes

https://www.mattrutherford.co.uk/laid-off-or-leaving/
4•speckx•24m ago•0 comments