frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Show HN: Fast and Quality Code Chunking with Chonkie

1•snyy•1y ago
Hi HN,

We’re Chonkie (https://github.com/chonkie-inc/chonkie) — we build open source tools that help split documents into meaningful chunks for use with AI models.

When you use LLMs over large documents or codebases, you often need to break them into smaller parts to fit the model’s context window. Our chunkers do this in a smart way: they preserve structure and meaning, so only the most relevant pieces are passed into the model. This reduces hallucinations, avoids confusion, and improves performance and accuracy.

Today we’re launching our Code Chunker — a fast, structure-aware way to break down source code into high-quality, token-aware chunks.

How it works:

(See the code: https://github.com/chonkie-inc/chonkie/blob/main/src/chonkie...)

Code Chunker uses tree-sitter (https://tree-sitter.github.io/tree-sitter/) to parse your code into an abstract syntax tree (AST). It then recursively merges and groups nodes in a way that respects both code structure and token limits.

It supports all languages that tree-sitter supports, and is designed to preserve formatting and semantics. Large functions or class definitions won’t be split in the middle of a block — instead, we dive recursively into the AST to produce clean, coherent chunks that fit your configured token budget.

What it’s useful for:

  - Embedding-based code search

  - RAG (retrieval-augmented generation) over codebases

  - Long-context analysis of code

  - Preparing repos for fine-tuning or pretraining
Try it out:

  - Open source package: https://docs.chonkie.ai/chunkers/code-chunker

  - Hosted playground (free with account): https://cloud.chonkie.ai
Happy Chonking!

Anthropic's founders and employees are now worth more than 45M families

https://bsky.app/profile/steverattner.bsky.social/post/3n2oxywvqy22v
2•doener•5m ago•0 comments

Solidcraft is the Fusion360 I always wanted

https://solidcraft.io/
1•olup•5m ago•0 comments

Show HN: Prove what your AI agent did – sign actions, verify in the browser

https://agentproofchain.com/try/
1•officialgovera•6m ago•1 comments

Why you should NOT run your own mail servers on your VPS

https://andreiclinciu.net/article/why-you-should-not-run-your-own-mail-servers-on-your-vps/
1•sammatilda•7m ago•1 comments

Foundation Models SDK for Python Documentation

https://apple.github.io/python-apple-fm-sdk/
1•janandonly•7m ago•0 comments

How Go Concurrency Works Under the Hood (Animated) [video]

https://www.youtube.com/watch?v=_kHcx7P2pMY
1•agentprince•8m ago•0 comments

Eighteen Things a Passkey Can Do

https://fenko.nz/blog/what-passkeys-can-do/
1•mosajjal•9m ago•0 comments

World Toilet Day

https://www.un.org/en/observances/toilet-day
1•andsoitis•10m ago•0 comments

Show HN: Ultra-Low Latency C++23 HFT Engine and Lock-Free Ring Buffers

https://github.com/grfdsasdvef/awesome-hft-cpp23-internals
1•snowdensay•11m ago•0 comments

The need for at least a third option

https://blog.ovalerio.net/archives/3359
1•dethos•11m ago•0 comments

Microsoft 365 outage of 12 and 13 November 2026

https://www.365fanatic.com/m365-outage-12-november-2026/
1•ahjones•12m ago•0 comments

Ask HN: Why aren't there more games where you use your body as a controller?

1•mristin•14m ago•0 comments

Beyond GQA: Characterizing Modern Attention Variants with ServingStudio

https://servingstudio.cs.washington.edu/blog/attention-variants/
1•matt_d•14m ago•0 comments

Emacs: Agent Shell 0.98 Updates

https://xenodium.com/agent-shell-0-98-updates
1•wmedrano•17m ago•0 comments

Bunkker ecosistema comercial offline sincronizado wifisync

https://github.com/ataca3000/bunnkker-e.c.o.s-baas
1•codematrix•17m ago•0 comments

Ask HN: OpenBSD 8.0 with GeForce GT 610?

1•o_-_o•17m ago•0 comments

The Quartz Page

http://www.quartzpage.de/index.html
1•AdamH12113•21m ago•0 comments

The End of the Beginning

https://www.robinsloan.com/lab/end-of-the-beginning/
1•tobr•29m ago•0 comments

Show HN: MCP-CrashTest – Multi-step stateful contract fuzzer for MCP servers

https://github.com/SnowyLumiere/mcp-crashtest
1•SnowyLumiere•32m ago•0 comments

SMAL

1•commoner_•32m ago•0 comments

NoEnigma – 15 Years of Independent Science and Technology Writing

https://www.noenigma.com/
1•jchacim•33m ago•0 comments

Longshore: A 36 GB/S Zero-Allocation I/O and Object Storage Layer in Rust

https://serviceradar.cloud/blog/ring-per-core-io-in-longshore
3•marvin-hansen•39m ago•0 comments

What Shaped Anthropic CEO Dario Amodei's Thinking on AI

https://www.nytimes.com/2026/11/05/technology/dario-amodei-anthropic-ai.html
1•andsoitis•41m ago•0 comments

Is it still worth building Coding Harness?

1•deepcover•41m ago•0 comments

Austria launches public consultation for its national AI agenda

https://www.kipppunkt.gv.at/
3•GranularRecipe•42m ago•0 comments

Who are Anthropic's seven co-founders, now worth $180B?

https://www.theguardian.com/technology/2026/nov/19/who-are-anthropic-founders
2•andsoitis•43m ago•0 comments

Implementing a tag-based restoration system for the Scalar docs sites

https://medium.com/scalar-engineering/implementing-a-tag-based-restoration-system-for-the-scalar-...
1•josh-scalar•43m ago•0 comments

Shift + 1234567890

https://koenvangilst.nl/lab/shift-1234567890
1•vnglst•43m ago•0 comments

Hostile Inference: Exploiting Docker Model Runner for Container Escapes

https://www.thezdi.com/blog/2026/11/17/hostile-inference-exploiting-docker-model-runner-for-conta...
1•ersinesen•45m ago•0 comments

Drone Hacking Part 2: Authorized ~_~ by Potensic ~_

https://neodyme.io/en/blog/drone_hacking_part_2/
2•ersinesen•46m ago•0 comments