frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Show HN: Fast and Quality Code Chunking with Chonkie

1•snyy•1y ago
Hi HN,

We’re Chonkie (https://github.com/chonkie-inc/chonkie) — we build open source tools that help split documents into meaningful chunks for use with AI models.

When you use LLMs over large documents or codebases, you often need to break them into smaller parts to fit the model’s context window. Our chunkers do this in a smart way: they preserve structure and meaning, so only the most relevant pieces are passed into the model. This reduces hallucinations, avoids confusion, and improves performance and accuracy.

Today we’re launching our Code Chunker — a fast, structure-aware way to break down source code into high-quality, token-aware chunks.

How it works:

(See the code: https://github.com/chonkie-inc/chonkie/blob/main/src/chonkie...)

Code Chunker uses tree-sitter (https://tree-sitter.github.io/tree-sitter/) to parse your code into an abstract syntax tree (AST). It then recursively merges and groups nodes in a way that respects both code structure and token limits.

It supports all languages that tree-sitter supports, and is designed to preserve formatting and semantics. Large functions or class definitions won’t be split in the middle of a block — instead, we dive recursively into the AST to produce clean, coherent chunks that fit your configured token budget.

What it’s useful for:

  - Embedding-based code search

  - RAG (retrieval-augmented generation) over codebases

  - Long-context analysis of code

  - Preparing repos for fine-tuning or pretraining
Try it out:

  - Open source package: https://docs.chonkie.ai/chunkers/code-chunker

  - Hosted playground (free with account): https://cloud.chonkie.ai
Happy Chonking!

2M 3D particle characters x2 fighting game in a single SVG

https://svgfpsaiagents.github.io/svgfpsgameAI/particlefighter.svg
1•zwanemakki•15s ago•0 comments

How much space is taken up by document versions?

https://eclecticlight.co/2026/11/21/how-much-space-is-taken-up-by-document-versions/
1•Brajeshwar•1m ago•0 comments

Playing GTA VI with a brain computer interface

https://twitter.com/neuralink/status/2123890949029818800
1•houqp•4m ago•0 comments

What is alternative grading? These TMU professors share how it works

https://www.ctvnews.ca/toronto/article/what-is-alternative-grading-these-tmu-professors-share-how...
1•amichail•7m ago•0 comments

The Future of Programming (2013)

https://worrydream.com/dbx/
1•zenincognito•8m ago•0 comments

Advent of Code 2026 starts in 10 days

https://www.reddit.com/r/adventofcode/comments/1xi8ltn/every_december_since_2015_ive_created_a_bu...
1•vismit2000•9m ago•0 comments

Vit B12-based air electrode for high-performance rechargeable zinc–air batteries

https://pubs.rsc.org/ta/article/14/50/34201/1280936/A-vitamin-B12-based-air-electrode-for-high
1•bookofjoe•9m ago•0 comments

Thenflation: A real life price guess game over the years

https://thenflation.com/
1•mvennekool•9m ago•1 comments

JEDEC Previews DDR6 Memory Standard: 2.35x More Bandwidth, 44% Narrower DIMMs

https://www.servethehome.com/jedec-previews-ddr6-memory-standard-2-35x-more-bandwidth-44-narrower...
1•ksec•9m ago•0 comments

MySQL Development as It Happens, Part 2 – Vector Indexes

https://blogs.oracle.com/mysql/mysql-development-as-it-happens-part-2-vector-indexes
1•ksec•10m ago•0 comments

Odin November 2026 Newsletter

https://odin-lang.org/news/newsletter-2026-11/
1•ksec•10m ago•0 comments

Qwen3.8 27B Just Became a Coding Monster (DaisyRose)

https://www.youtube.com/watch?v=bwsSnLM3YFI
1•lroe•11m ago•0 comments

Battery storage in Great Britain: how a grid battery makes money

https://a115.co.uk/battery-storage-revenues/
1•a115ltd•13m ago•0 comments

Branchi – Open-Source Git Client for Windows and Linux (Rust and Flutter)

https://github.com/mbpcoder/Branchi
1•mbpcoder•15m ago•0 comments

Armiger – a local ONNX torrent classifier for Bitmagnet and Rats Search

https://github.com/spencernorton/armiger
1•bilegeek•19m ago•0 comments

Ask HN: Best books you've read this year (2026)?

1•learner_yearner•21m ago•0 comments

Free System Design Course (Visual Learners)

https://systems-design.aiengineer.co/
3•owlewis•22m ago•0 comments

'Positive review only': Researchers hide AI prompts in papers Asia

https://asia.nikkei.com/business/technology/artificial-intelligence/positive-review-only-research...
1•colinprince•25m ago•0 comments

Apollo, Goldman lead $44B SpaceX GPU debt financing

https://www.reuters.com/business/media-telecom/apollo-goldman-lead-44-billion-spacex-gpu-debt-fin...
2•jeffufl•26m ago•0 comments

"Everything app" vs. "Context layer"

https://twitter.com/mdubakov/status/2124051223921799278
2•tablet•27m ago•0 comments

Show HN: rumblex – word timestamps and subtitles from Rumble-2-Flash's CTC head

https://github.com/EdwardGlockner/rumblex
1•edwardglockner•31m ago•0 comments

Fight over fossil fuels leaves COP31 climate talks in deadlock

https://www.bbc.com/news/articles/c6ly5w40eg07o
2•blondie9x•33m ago•0 comments

Wielding the written word: How to pitch and publish science prose

https://physicstoday.aip.org/features/wielding-the-written-word-how-to-pitch-and-publish-science-...
2•colinprince•36m ago•0 comments

What do Deutsche Bank leaks reveal about Trump, Epstein and Russia?

https://www.aljazeera.com/news/2026/11/18/what-do-deutsche-bank-leaks-reveal-about-trump-epstein-...
5•enoint•37m ago•0 comments

Show HN: agent-mcts – Monte Carlo tree search on top of Claude Code

https://github.com/natsu0529/agent-mcts
3•natsu529•39m ago•0 comments

Show HN: Accounting built like Git, with files in your own storage

https://github.com/spooky6G/bobalice_books
1•oskar77•39m ago•0 comments

The Machine in the Back of the Book

https://levships.com/books/the-machine-in-the-back-of-the-book/
1•AlexeyBrin•40m ago•0 comments

Inkling-2-Mini-GGUF

https://huggingface.co/unsloth/Inkling-2-Mini-GGUF
1•doener•40m ago•0 comments

Show HN: My encrypted agent OS gives OpenClaw and Hermes an Unreal Engine avatar

https://github.com/oscova/oscova-agents
1•driftvoy•41m ago•0 comments

The Kafkaian Agent Swarm That Locked Out My Cofounder

https://simeoncampos.substack.com/p/on-the-kafkaian-agent-swarm-that
1•fdb•42m ago•0 comments