frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Show HN: Fast and Quality Code Chunking with Chonkie

1•snyy•1y ago
Hi HN,

We’re Chonkie (https://github.com/chonkie-inc/chonkie) — we build open source tools that help split documents into meaningful chunks for use with AI models.

When you use LLMs over large documents or codebases, you often need to break them into smaller parts to fit the model’s context window. Our chunkers do this in a smart way: they preserve structure and meaning, so only the most relevant pieces are passed into the model. This reduces hallucinations, avoids confusion, and improves performance and accuracy.

Today we’re launching our Code Chunker — a fast, structure-aware way to break down source code into high-quality, token-aware chunks.

How it works:

(See the code: https://github.com/chonkie-inc/chonkie/blob/main/src/chonkie...)

Code Chunker uses tree-sitter (https://tree-sitter.github.io/tree-sitter/) to parse your code into an abstract syntax tree (AST). It then recursively merges and groups nodes in a way that respects both code structure and token limits.

It supports all languages that tree-sitter supports, and is designed to preserve formatting and semantics. Large functions or class definitions won’t be split in the middle of a block — instead, we dive recursively into the AST to produce clean, coherent chunks that fit your configured token budget.

What it’s useful for:

  - Embedding-based code search

  - RAG (retrieval-augmented generation) over codebases

  - Long-context analysis of code

  - Preparing repos for fine-tuning or pretraining
Try it out:

  - Open source package: https://docs.chonkie.ai/chunkers/code-chunker

  - Hosted playground (free with account): https://cloud.chonkie.ai
Happy Chonking!

Markdown reader with images, LaTeX formulas rendering, written in C with SDL2

https://github.com/vincenzoaurelio/Platessa
1•vincenzoaurelio•27s ago•1 comments

You're riding a faster horse

https://friendlyai.dev/blog/you-are-riding-a-fast-horse/
1•FriendlyMike•1m ago•0 comments

No Gas, No Chips: Mapping the Choke Points Behind Chip Manufacturing

https://www.aei.org/research-products/report/no-gas-no-chips-mapping-the-choke-points-behind-chip...
1•voxleone•3m ago•0 comments

Ratwires – The Graffiti Wall of the Internet

https://ratwire.space/
1•fdgsdfsdf•3m ago•0 comments

The End of the No-Brainer AI Trade

https://www.wsj.com/finance/stocks/the-month-that-rocked-the-ai-trade-and-rewired-wall-street-f33...
1•thm•3m ago•0 comments

Waves that clear waste from the brain during sleep are impaired in older adults

https://news.mit.edu/2026/waste-clearing-waves-in-brain-during-sleep-are-impaired-in-older-adults...
1•gmays•3m ago•0 comments

Lemon Squeezy is winding down

https://docs.lemonsqueezy.com/migration
1•martin-adams•3m ago•1 comments

Show HN: Grapim, open-source AI code review that runs on your own model key

https://github.com/grapim/grapim
1•alexanderko104•5m ago•0 comments

New Unblocked Games Site

https://poly2.benlaunches.icu
1•notam•5m ago•0 comments

Quark: Our whole training stack, end to end, for $4,200

https://empero.org/writing/quark
2•gmays•7m ago•0 comments

Linktwos finds who you need every day and who to meet at conferences

https://linktwos.com/
2•tomhartleymills•7m ago•0 comments

Remote Access to American Super Intelligence Technology

https://media.bis.gov/press-release/bis-issues-rule-addressing-remote-access-american-super-intel...
1•sharpshadow•8m ago•0 comments

Robert Pattinson confirmed as new James Bond

https://news.sky.com/story/robert-pattinson-confirmed-as-new-james-bond-13612738
1•austinallegro•8m ago•0 comments

Janet Reno was talking about my friend Coolio (computer criminal)

https://michael.bacarella.com/2026/11/16/janet-reno-vs-my-friend-coolio/
1•mbac32768•9m ago•0 comments

Entropy Hunters: Random Numbers and Chaos Machines

https://pod.link/1879625858/episode/QnV6enNwcm91dC0xOTUyNDE0MA
1•dbouquin•9m ago•0 comments

Show HN: Make a custom deck of playing cards with your family and friend's faces

https://facevaluepress.com/
1•mitenko•10m ago•1 comments

Substantial number of fund managers have failed to outperform the market (2025)

https://www.ajbell.co.uk/news/shocking-number-fund-managers-have-failed-outperform-market
2•toomuchtodo•11m ago•0 comments

Some places have banned AI glasses, and the list just keeps growing

https://www.engadget.com/2313707/ai-glasses-banned-places-list-keeps-growing/
1•drodio•11m ago•0 comments

Looking for a remote CTO for my micro SaaS ($600K ARR, $250k Profit)

1•smurf9852•11m ago•1 comments

Cursor: An update on our recent security incident

https://cursor.com/blog/security-incident-update
1•drodio•11m ago•0 comments

Gootloader Malware Using SEO Poisoning and Compromised Websites to Gain Access [pdf]

https://www.ic3.gov/CSA/2026/261116.pdf
1•saikatsg•13m ago•0 comments

Rosalind Workbench: Empowering every scientist to be their own research team

https://developers.openai.com/blog/rosalind-workbench
1•gmays•13m ago•0 comments

Threat Overview of Artex Autonomous Penetration Testing Framework [pdf]

https://www.ic3.gov/CSA/2026/261113.pdf
1•saikatsg•14m ago•0 comments

Artex – AI autonomous penetration testing system

https://github.com/BaaRai/ARTEX
1•saikatsg•15m ago•0 comments

Psychic: Like fzf but learns from your actions

https://klauria.tech/psychic/
2•articulatepang•16m ago•0 comments

In Search of the Deep Life

https://calnewport.com/in-search-of-the-deep-life/
1•samizdis•17m ago•0 comments

Suspicious Traffic Claiming Referral from Aocr.org

https://vale.rocks/micros/20261110-1531
2•OuterVale•17m ago•0 comments

Robert Pattinson is the new James Bond

https://www.aboutamazon.com/news/entertainment/james-bond-robert-pattinson-cast-amazon-mgm-film
1•HelloUsername•17m ago•0 comments

Cgo's pointer check is all over the place

https://theconsensus.dev/p/2026/11/15/cgos-pointer-check.html
1•shraddhaag•18m ago•0 comments

My 17 Heroes – My PhD Students (Bill Wadge)

https://billwadge.com/2026/11/15/my-17-heroes-my-phd-students/
1•herodotus•18m ago•0 comments