frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Show HN: Fast and Quality Code Chunking with Chonkie

1•snyy•1y ago
Hi HN,

We’re Chonkie (https://github.com/chonkie-inc/chonkie) — we build open source tools that help split documents into meaningful chunks for use with AI models.

When you use LLMs over large documents or codebases, you often need to break them into smaller parts to fit the model’s context window. Our chunkers do this in a smart way: they preserve structure and meaning, so only the most relevant pieces are passed into the model. This reduces hallucinations, avoids confusion, and improves performance and accuracy.

Today we’re launching our Code Chunker — a fast, structure-aware way to break down source code into high-quality, token-aware chunks.

How it works:

(See the code: https://github.com/chonkie-inc/chonkie/blob/main/src/chonkie...)

Code Chunker uses tree-sitter (https://tree-sitter.github.io/tree-sitter/) to parse your code into an abstract syntax tree (AST). It then recursively merges and groups nodes in a way that respects both code structure and token limits.

It supports all languages that tree-sitter supports, and is designed to preserve formatting and semantics. Large functions or class definitions won’t be split in the middle of a block — instead, we dive recursively into the AST to produce clean, coherent chunks that fit your configured token budget.

What it’s useful for:

  - Embedding-based code search

  - RAG (retrieval-augmented generation) over codebases

  - Long-context analysis of code

  - Preparing repos for fine-tuning or pretraining
Try it out:

  - Open source package: https://docs.chonkie.ai/chunkers/code-chunker

  - Hosted playground (free with account): https://cloud.chonkie.ai
Happy Chonking!

The Cottage Industry That Is the AI Training Data Market

https://twitter.com/DeepInsightLabs/status/2123721278112911860
1•nlpnerd•20s ago•0 comments

Attackers hid M365 takeovers with inbox rules named "a", "cc" and " "

https://www.reco.ai/blog/three-characters-no-conditions-an-m365-session-theft-intrusion-that-move...
1•daretograsp•1m ago•0 comments

Show HN: Heartpad – send a heartbeat you can feel on a MacBook trackpad

https://heartpad.app/
1•vikasghodke•4m ago•0 comments

The History of Logic [video]

https://www.youtube.com/watch?v=-wgMQcip24s
1•nill0•5m ago•0 comments

A Stroll Along the South Bank, London

https://brajeshwar.com/misc/london-southbank/
1•Brajeshwar•6m ago•0 comments

New report documents authoritarian use of technology across the globe

https://edri.org/our-work/new-report-documents-authoritarian-use-of-technology-across-the-globe/
1•DeepLogin•8m ago•0 comments

Why do tech leaders sound so illogical and off-putting?

https://www.theatlantic.com/technology/2026/11/silicon-valley-model-collapse/689505/
2•pseudolus•9m ago•1 comments

Serverless, Cheaterless Magic: The Gathering

https://chiplis.com/blog/serverless-cheaterless-magic
1•khaliostr•9m ago•0 comments

TSMC Explains Semiconductor Chips in 3 Levels – TSMC × Wired [video]

https://www.youtube.com/watch?v=1DVAs2fKCmQ
2•vismit2000•9m ago•0 comments

Intuit to sell Mailchimp to Bending Spoons for $3.68B

https://www.cnbc.com/2026/11/19/intuit-mailchimp-sale-bending-spoons.html
4•pseudolus•14m ago•0 comments

Pill to treat pancreatic cancer hailed as 'biggest breakthrough in decades'

https://www.theguardian.com/society/2026/nov/20/pill-pancreatic-cancer-breakthrough-daraxonrasib
2•Gupie•14m ago•0 comments

Lenders aim to bar SpaceX from launching $40B of chips into orbit

https://www.ft.com/content/33b5643c-2957-4d42-9116-5e34a3c54e29
3•ryzvonusef•14m ago•2 comments

Same Streets, Different Empires: The Hidden Narrative of Map Copyright

https://blogs.loc.gov/maps/2026/11/same-streets-different-empires/
1•bryanrasmussen•18m ago•0 comments

How to tell if a Reddit post is AI: which signs got more common

https://kissmypost.com/research/how-to-tell-if-a-reddit-post-is-ai
1•sibxog-6Zotpy-g•19m ago•0 comments

How much of Reddit is bots?

https://kissmypost.com/research/how-much-of-reddit-is-bots
2•sibxog-6Zotpy-g•19m ago•0 comments

The Use of AI-Based Tools to Find Bugs in Open Source Software

https://www.haproxy.com/blog/on-the-use-of-ai-based-tools-to-find-bugs-in-open-source-software
1•hosteur•22m ago•0 comments

Bending Spoons has entered into a definitive agreement to acquire Mailchimp

https://www.businesswire.com/news/home/20261119179325/en/Bending-Spoons-has-entered-into-a-defini...
1•sdoering•25m ago•0 comments

Chrome Extension That Shows Everyone's TC on LinkedIn

https://chromewebstore.google.com/detail/paypeek-total-comp-estima/aenhepbomobiielgagkkaagkbjaljobj
1•Donkwagonhan•25m ago•0 comments

Chaos, a NTP server in OCaml as an unikernel

https://blog.robur.coop/articles/2026-11-19-chaos.html
1•dinosaure•27m ago•0 comments

Best Study Abroad Consultants in Kerala – Start Your Global Education Journey

1•agasto•30m ago•0 comments

Dutch privacy watchdog AP advises consumers not to purchase smart glasses

https://www.reuters.com/business/media-telecom/dutch-privacy-watchdog-ap-advises-consumers-not-pu...
2•giuliomagnifico•30m ago•0 comments

Show HN: Interviewer assistant – AI open source

https://github.com/blackarck/intaiassist
1•blackarck•30m ago•0 comments

Kokoro Yoruba – Native Tonal Yoruba TTS on StyleTTS2 / Kokoro-82M

https://github.com/Akintunde102/kokoro-yoruba
1•Akintunde102•31m ago•0 comments

Vim Classic 8.3.1 Released

https://vim-classic.org/news/vim-8.3.1-released.html
5•birdculture•33m ago•0 comments

Telegram now works on the "both too big and too small" iPhone

https://twitter.com/durov/status/2123542555321397660
1•zdkaster•33m ago•0 comments

Block Anything on YouTube

3•devanshvarshney•35m ago•0 comments

The Most Powerful Men in America Are Eating Only Steak and Sauerkraut. Could I?

https://www.nytimes.com/2026/08/04/magazine/trump-rfk-steak-kimchi-diet.html
2•Kaibeezy•40m ago•1 comments

The Making of Oxygene

https://jeanmicheljarre.com/aaa/the-making-of-oxygene
5•hggh•40m ago•1 comments

PowerShell 7 is coming to Windows

https://devblogs.microsoft.com/powershell/powershell-7-coming-to-windows/
1•AndrewDucker•45m ago•0 comments

Working as Intended [video]

https://www.youtube.com/watch?v=anwKw6oWsfg
1•stefanv•45m ago•0 comments