frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Show HN: Fast and Quality Code Chunking with Chonkie

1•snyy•1y ago
Hi HN,

We’re Chonkie (https://github.com/chonkie-inc/chonkie) — we build open source tools that help split documents into meaningful chunks for use with AI models.

When you use LLMs over large documents or codebases, you often need to break them into smaller parts to fit the model’s context window. Our chunkers do this in a smart way: they preserve structure and meaning, so only the most relevant pieces are passed into the model. This reduces hallucinations, avoids confusion, and improves performance and accuracy.

Today we’re launching our Code Chunker — a fast, structure-aware way to break down source code into high-quality, token-aware chunks.

How it works:

(See the code: https://github.com/chonkie-inc/chonkie/blob/main/src/chonkie...)

Code Chunker uses tree-sitter (https://tree-sitter.github.io/tree-sitter/) to parse your code into an abstract syntax tree (AST). It then recursively merges and groups nodes in a way that respects both code structure and token limits.

It supports all languages that tree-sitter supports, and is designed to preserve formatting and semantics. Large functions or class definitions won’t be split in the middle of a block — instead, we dive recursively into the AST to produce clean, coherent chunks that fit your configured token budget.

What it’s useful for:

  - Embedding-based code search

  - RAG (retrieval-augmented generation) over codebases

  - Long-context analysis of code

  - Preparing repos for fine-tuning or pretraining
Try it out:

  - Open source package: https://docs.chonkie.ai/chunkers/code-chunker

  - Hosted playground (free with account): https://cloud.chonkie.ai
Happy Chonking!

Canvas Dx, an AI app that diagnoses autism [video]

https://vimeo.com/791026418
1•illidan1000•39s ago•1 comments

Americans misled on Social Security for decades as clock counts down on benefits

https://www.foxnews.com/politics/americans-misled-social-security-decades-clock-counts-benefits-e...
1•dataflow•52s ago•0 comments

In France and Germany, the state backs one startup round in three

https://justraised.io/blog/state-backs-one-startup-round-in-three
1•jrbayguade•3m ago•0 comments

Jj-musubi: live sync for Jujutsu

https://www.serpentine.com/posts/2026/announcing-jj-musubi/
1•bos•4m ago•0 comments

Show HN: Toggles, a game based on analyzing data

https://toggles.codewalkers.net
1•mattw2121•5m ago•0 comments

Growing as an Engineer in a World of AI

https://nlopes.dev/writing/growing-as-an-engineer-in-a-world-of-ai
1•fagnerbrack•7m ago•0 comments

Apple Was Late to the Video Podcast Boom. Now It's Trying to Catch Up

https://www.nytimes.com/2026/11/16/business/media/apple-tv-podcasts-youtube.html
2•bookofjoe•9m ago•1 comments

To rank the biggest supercomputers, we're gonna need a bigger bench

https://www.theregister.com/hpc/2026/11/18/to-rank-the-worlds-biggest-supercomputers-were-gonna-n...
1•rbanffy•9m ago•0 comments

Q.Ant NPU Gen2 Photonic Processor

https://thequantuminsider.com/2026/11/17/q-ant-demonstrates-hybrid-photonic-gpu-system-for-langua...
1•treebeard901•10m ago•1 comments

Show HN: Phrack-TUI – A terminal reader for the Phrack archive

https://github.com/krmbzds/phrack-tui/tree/master
1•krmbzds•11m ago•0 comments

Scientists Thought This Amphibious Mouse Was Extinct. It's Reappeared

https://www.nationalgeographic.com/animals/article/ethiopia-water-mouse-extinct-rediscovery
1•spankibalt•12m ago•0 comments

GTA VI is a cynical and sometimes shallow joyride through the American psyche

https://www.cbc.ca/news/entertainment/grand-theft-auto-vi-review-9.7425005
2•uladzislau•14m ago•1 comments

CodeRabbit Verify: Agentic application testing and QA

https://www.coderabbit.ai/blog/introducing-coderabbit-verify
1•smb06•19m ago•1 comments

Amazon vows to fight the company's first NA union contract at a CAN warehouse

https://apnews.com/article/amazon-unifor-union-workers-1e0626c31c857ad54aa9b20d8c83b54d
2•Teever•19m ago•0 comments

The Life of a Composer

https://composer.pablostanley.com/#1
1•alexnew•19m ago•0 comments

Chainguard Introduces Free Starter and $500 Team Plans

https://devops.com/chainguard-introduces-free-starter-and-500-team-plans
1•CrankyBear•20m ago•0 comments

Nvidia Tells Employees That Company Is Moving on from Huang Era

https://www.bloomberg.com/news/articles/2026-11-18/nvidia-tells-employees-that-company-is-moving-...
3•TheAlchemist•20m ago•0 comments

Grand Theft Auto VI Critic Reviews

https://www.metacritic.com/game/grand-theft-auto-vi/critic-reviews/
3•TechTechTech•21m ago•2 comments

Play drums with your keyboard

https://plastic.audio/drum-pad/
1•madbyte•22m ago•0 comments

The social use of meow in the interaction between cats and humans

https://www.sciencedirect.com/science/article/abs/pii/S0003347226002940?via%3Dihub=
1•ohjeez•22m ago•0 comments

Harvard study estimates magnitude of US data center pollution - climate talks

https://www.nbcnews.com/tech/tech-news/harvard-study-estimates-magnitude-us-data-center-pollution...
1•Goodliff•22m ago•0 comments

Energy Wins Driving Optical Computing AI

https://spectrum.ieee.org/optical-computing-ai
1•rbanffy•22m ago•0 comments

Show HN: A 149M model handles 70% of phone-agent steps locally, Claude the rest

https://github.com/chayworks/milo-pointer
1•chaitanyav33•22m ago•0 comments

Show HN: A forkable folk song; each recording holds the code that sang it

https://now.itsby.you/
1•sorghum•23m ago•0 comments

HubSpot Lost 80% of Its Traffic – CMO Explains Why Revenue Kept Growing

https://www.searchenginejournal.com/hubspot-lost-80-of-its-traffic-cmo-kipp-bodnar-explains-why-r...
1•gscott•24m ago•0 comments

Java4Shell: When base64 decoding leads to remote code execution?

https://lemire.me/blog/2026/11/18/java4shell-when-base64-decoding-leads-to-remote-code-execution/
1•josephscott•25m ago•0 comments

Anthropic Prices IPO at $83 a Share

https://www.wsj.com/finance/stocks/anthropic-prices-ipo-at-83-a-share-0290494c
3•dcgudeman•25m ago•0 comments

Google Buys Worker Data from Spirit Airlines in a New Market to Feed AI

https://www.nytimes.com/2026/11/18/business/economy/google-spirit-employee-data-ai.html
3•haunter•26m ago•1 comments

Show HN: InfraSim, a system design simulator game for iOS and Android

https://petalformllc.com/apps/infrasim/
1•petalform•26m ago•0 comments

Hailo is now part of Microchip

https://hailo.ai/company-overview/newsroom/news/continuing-to-build-with-hailo/
1•geerlingguy•27m ago•0 comments