frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Show HN: Fast and Quality Code Chunking with Chonkie

1•snyy•1y ago
Hi HN,

We’re Chonkie (https://github.com/chonkie-inc/chonkie) — we build open source tools that help split documents into meaningful chunks for use with AI models.

When you use LLMs over large documents or codebases, you often need to break them into smaller parts to fit the model’s context window. Our chunkers do this in a smart way: they preserve structure and meaning, so only the most relevant pieces are passed into the model. This reduces hallucinations, avoids confusion, and improves performance and accuracy.

Today we’re launching our Code Chunker — a fast, structure-aware way to break down source code into high-quality, token-aware chunks.

How it works:

(See the code: https://github.com/chonkie-inc/chonkie/blob/main/src/chonkie...)

Code Chunker uses tree-sitter (https://tree-sitter.github.io/tree-sitter/) to parse your code into an abstract syntax tree (AST). It then recursively merges and groups nodes in a way that respects both code structure and token limits.

It supports all languages that tree-sitter supports, and is designed to preserve formatting and semantics. Large functions or class definitions won’t be split in the middle of a block — instead, we dive recursively into the AST to produce clean, coherent chunks that fit your configured token budget.

What it’s useful for:

  - Embedding-based code search

  - RAG (retrieval-augmented generation) over codebases

  - Long-context analysis of code

  - Preparing repos for fine-tuning or pretraining
Try it out:

  - Open source package: https://docs.chonkie.ai/chunkers/code-chunker

  - Hosted playground (free with account): https://cloud.chonkie.ai
Happy Chonking!

Tesla goes all-in on the Cybercab with upcoming €3B Berlin gigafactory expansion

https://techcrunch.com/2026/11/16/tesla-goes-all-in-on-the-cybercab-with-upcoming-e3b-berlin-giga...
1•dvaplima•1m ago•1 comments

Grand Theft Auto 6 is a masterpiece, say critics

https://www.bbc.co.uk/news/articles/cw8r75xm8v4po
1•iamben•1m ago•0 comments

Show HN: RailCall – Local-first agent runtime with approvals and signed receipts

https://railcall.ai/
1•nickyocean•3m ago•0 comments

Windows PowerShell 5.1 Deprecation

https://support.microsoft.com/en-gb/servicing/os/windows/docs/2026/11/kb5137172-windows-powershel...
1•delta_p_delta_x•4m ago•1 comments

Songcode is doing music generation with code

https://songco.de/
1•jakehandy•4m ago•1 comments

The future of financial reporting is not a chart

https://www.brex.com/journal/articles/the-future-of-financial-reporting-is-not-a-chart
1•skadamat•4m ago•0 comments

Why I'm Turning Off Comments on My Blog

https://www.lireo.com/turning-off-comments/
2•speckx•4m ago•0 comments

Show HN: Herdr mod that makes managing agents like playing Runescape

https://github.com/davidvgilmore/RuneSpace
1•davidvgilmore•9m ago•0 comments

LinkedIn Is So Smart

1•mring33621•10m ago•0 comments

The seven Anthropic founders who are about to become multi-billionaires

https://www.bbc.com/news/articles/cmd7159vlr85o
1•saikatsg•10m ago•0 comments

SF bay woman, who started a cacao farm in Panama, was murdered

https://newsroompanama.com/2026/11/15/double-homicide-in-chiriqui/
3•Caarticles•10m ago•1 comments

Ask HN: Windows malware removal in Bay Area

1•HoldOnAMinute•11m ago•0 comments

Show HN: HimitsuShell – Shell scripts that evade kernel tracing

https://github.com/HimitsuShell/HimitsuShell
1•mushstory•11m ago•0 comments

RIP Ken St. Andre (1947–2026)

http://grognardia.blogspot.com/2026/11/rip-ken-st-andre-19472026.html
2•speckx•13m ago•0 comments

DaisyRose-27B: 1,200-task RL took Qwen3.8 from 5.6% to 12% on Terminal-Bench 4.1

https://www.radixark.com/blog/daisyrose-27b
3•de_sty•14m ago•0 comments

Sellas marks 9-month survival benefit in late-stage leukemia vaccine study

https://www.fiercebiotech.com/biotech/sellas-marks-nine-month-survival-benefit-late-stage-leukemi...
1•toomuchtodo•14m ago•0 comments

GitHub Redesign (2019)

https://tonsky.me/blog/github-redesign/
2•birdculture•15m ago•0 comments

Fed Chair: Inflation is "headed in the wrong direction, and it's spreading out"

https://www.cnbc.com/2026/11/18/federal-reserve-governor-christopher-waller.html
4•novaRom•16m ago•2 comments

C64 Basic Dungeon Crawler: Fog of War and the Need for Speed (Part 13)

https://retrogamecoders.com/c64-basic-dungeon-part13/
2•ibobev•17m ago•0 comments

Eclipse Foundation (Adoptium Temurin) Omitted from OpenJDK Vulnerability Group

https://github.com/adoptium/temurin/issues/326
1•JacobHenner•18m ago•1 comments

TCBP-JOBS: GPU-Accelerated Quantum Error Correction

https://arxiv.org/abs/2611.21321
2•jonbaer•18m ago•0 comments

Live streaming model evals executing real DevOps tasks

https://opendevops.ai
1•dannypdx•18m ago•1 comments

I don't use an AI assistant for writing

https://www.suramya.com/blog/2026/11/why-i-dont-use-ai-assistant-for-writing/
1•speckx•18m ago•0 comments

Girl, 6, is killed in gene-editing trial in China as scientist is banned

https://www.the-express.com/news/science/220008/scientist-girl-6-dies-trial
2•randycupertino•19m ago•0 comments

Can LLMs Execute Algorithms?

https://vkcr.dev/blog/02-gcd/
2•vkgcr•20m ago•0 comments

The Amiga birthed some truly phenomenal games

https://metro.co.uk/2026/11/16/thea1200-retro-games-interview-the-amiga-birthed-truly-phenomenal-...
1•ibobev•21m ago•0 comments

The Silent Consensus

https://openai.com/index/the-silent-consensus/
1•md224•21m ago•0 comments

Show HN: Vibes Matrix helps you make complex life decisions

https://vibesmatrix.com/
1•culturengine•21m ago•1 comments

Ask HN: Where are you following security vulnerabilities?

2•ungawatkt•22m ago•1 comments

'Situational Awareness 2.0': Leopold Aschenbrenner plots comeback

https://www.ft.com/content/1fcbc6ea-1be3-44d2-b564-8ebf91181639
2•JumpCrisscross•23m ago•0 comments