frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Show HN: Fast and Quality Code Chunking with Chonkie

1•snyy•1y ago
Hi HN,

We’re Chonkie (https://github.com/chonkie-inc/chonkie) — we build open source tools that help split documents into meaningful chunks for use with AI models.

When you use LLMs over large documents or codebases, you often need to break them into smaller parts to fit the model’s context window. Our chunkers do this in a smart way: they preserve structure and meaning, so only the most relevant pieces are passed into the model. This reduces hallucinations, avoids confusion, and improves performance and accuracy.

Today we’re launching our Code Chunker — a fast, structure-aware way to break down source code into high-quality, token-aware chunks.

How it works:

(See the code: https://github.com/chonkie-inc/chonkie/blob/main/src/chonkie...)

Code Chunker uses tree-sitter (https://tree-sitter.github.io/tree-sitter/) to parse your code into an abstract syntax tree (AST). It then recursively merges and groups nodes in a way that respects both code structure and token limits.

It supports all languages that tree-sitter supports, and is designed to preserve formatting and semantics. Large functions or class definitions won’t be split in the middle of a block — instead, we dive recursively into the AST to produce clean, coherent chunks that fit your configured token budget.

What it’s useful for:

  - Embedding-based code search

  - RAG (retrieval-augmented generation) over codebases

  - Long-context analysis of code

  - Preparing repos for fine-tuning or pretraining
Try it out:

  - Open source package: https://docs.chonkie.ai/chunkers/code-chunker

  - Hosted playground (free with account): https://cloud.chonkie.ai
Happy Chonking!

Show HN: Fluxtail – Centralized log search and Live Tail

https://indielaunches.com/projects/prj_diyrnmfunvob-fluxtail
2•anabes•3m ago•0 comments

US to curb China's access to AI chips

https://www.reuters.com/legal/litigation/us-issues-rule-protect-advanced-ai-chips-being-accessed-...
2•geox•3m ago•0 comments

When Rounding Breaks Machine Learning

https://www.robertj1.com/ai4science/rounding-bugs-in-ml-formulas/
1•sthottingal•3m ago•1 comments

The Kottke.org Video Closet

https://kottke.org/26/11/the-kottkeorg-video-closet
1•colinprince•5m ago•0 comments

TypeScript apps running natively in embedded devices

https://github.com/geastack/demos
1•arbayi•5m ago•0 comments

Unkernel: Linux kernel in WASM

https://unkernel.dev
1•dvrp•5m ago•0 comments

External Sandboxes for Coding Agents

https://blog.primeradiant.com/external-sandboxes-for-coding-agents/
1•emschwartz•7m ago•0 comments

Extending separation logic to verify the xv6 OS kernel on RISC-V with AI agents

https://arxiv.org/abs/2609.04043
1•chubot•11m ago•0 comments

Nvidia faces daunting challenge of replacing CEO Jensen Huang

https://www.washingtonpost.com/technology/2026/11/14/nvidia-faces-daunting-challenge-replacing-ce...
1•gmays•12m ago•0 comments

Show HN: Woolly, a local mem0-compatible memory API for AI agents

https://usewoolly.com/
1•rjcuff•13m ago•0 comments

AI Agents Can't Talk

https://cacm.acm.org/blogcacm/ai-agents-cant-talk/
1•abhilash539•17m ago•0 comments

A game cartridge peripheral for Dreamcast

https://github.com/ThroatyMumbo/Dreamcart
1•nticompass•20m ago•0 comments

Niewz – a native RSS reader for iPhone, iPad and Mac

https://apps.apple.com/us/app/niewz/id6834017988
1•romainrou•26m ago•0 comments

Brad Feld – What Do You Want to Do Tonight?

https://feld.com/archives/2026/11/what-do-you-want-to-do-tonight/
1•rmason•29m ago•1 comments

IPTV USA Subscription – IPTV USA – 50k Channels and 4K Streaming

https://www.nhl.com/goldenknights/news/vegas-jr-golden-knights-announce-teams-tryout-information-...
1•saraapah•34m ago•2 comments

Scientists Are Stumped by This Object in Space

https://www.nytimes.com/2026/11/16/science/1998-ky26-asteroid-comet.html
1•reaperducer•35m ago•0 comments

The Emulation Frontier

https://alexlitzenberger.com/blog/the_emulation_frontier
1•alexlitz•36m ago•0 comments

WSJ: Jensen Huang was a difficult boss

https://www.wsj.com/business/put-downs-parties-and-a-royal-family-the-culture-inside-jensen-huang...
2•onmysofa•41m ago•2 comments

My Father Gave the Internet Declaration of Independence; Time for a Second Draft

https://www.rollingstone.com/culture/culture-commentary/amelia-rose-john-perry-barlow-internet-de...
1•the-mitr•47m ago•0 comments

Show HN: Sip pacer to help you drink slower

https://apps.apple.com/us/app/pour-decision-drink-less/id6499468185
1•OzBuilds•48m ago•0 comments

The WeirdML Benchmark (2025)

https://www.lesswrong.com/posts/LfQCzph7rc2vxpweS/introducing-the-weirdml-benchmark
1•peter_d_sherman•49m ago•1 comments

Everything You Wanted to Know About LLM Post-Training [video]

https://www.youtube.com/watch?v=LVXtFnEbNU0
1•gmays•55m ago•0 comments

Nvidia Announces Rubin HPC

https://blogs.nvidia.com/blog/rubin-hpc/
2•jonbaer•55m ago•0 comments

Show HN: Pulsar – cross-harness agent to agent communication and messaging

https://github.com/yocontra/pulsar
1•contrahax•55m ago•0 comments

Show HN: Find out how cooked your webapp is

https://genclass.dev/scan
1•Karanvir1729•56m ago•0 comments

News from more than one source

https://www.simplenewss.com/
1•jream1991•57m ago•1 comments

Mathematical Modeling Basics [pdf]

https://matmod.ch/lpl/doc/modelbook.pdf
2•the-mitr•58m ago•0 comments

US Data Centers' Carbon Footprint Rivals Small European Countries

https://www.barrons.com/news/us-data-centers-carbon-footprint-rivals-small-european-countries-0c4...
2•melling•1h ago•0 comments

Nvidia's top lawyer agrees to testify before Congress

https://www.politico.com/news/2026/11/16/nvidia-top-lawyer-testify-congress-01168851
3•JumpCrisscross•1h ago•0 comments

123456' has been the most popular password for 7 out of the last 8 years

https://www.theregister.com/security/2026/11/16/123456-has-been-the-worlds-most-popular-password-...
2•madihaa•1h ago•1 comments