frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Show HN: Fast and Quality Code Chunking with Chonkie

1•snyy•1y ago
Hi HN,

We’re Chonkie (https://github.com/chonkie-inc/chonkie) — we build open source tools that help split documents into meaningful chunks for use with AI models.

When you use LLMs over large documents or codebases, you often need to break them into smaller parts to fit the model’s context window. Our chunkers do this in a smart way: they preserve structure and meaning, so only the most relevant pieces are passed into the model. This reduces hallucinations, avoids confusion, and improves performance and accuracy.

Today we’re launching our Code Chunker — a fast, structure-aware way to break down source code into high-quality, token-aware chunks.

How it works:

(See the code: https://github.com/chonkie-inc/chonkie/blob/main/src/chonkie...)

Code Chunker uses tree-sitter (https://tree-sitter.github.io/tree-sitter/) to parse your code into an abstract syntax tree (AST). It then recursively merges and groups nodes in a way that respects both code structure and token limits.

It supports all languages that tree-sitter supports, and is designed to preserve formatting and semantics. Large functions or class definitions won’t be split in the middle of a block — instead, we dive recursively into the AST to produce clean, coherent chunks that fit your configured token budget.

What it’s useful for:

  - Embedding-based code search

  - RAG (retrieval-augmented generation) over codebases

  - Long-context analysis of code

  - Preparing repos for fine-tuning or pretraining
Try it out:

  - Open source package: https://docs.chonkie.ai/chunkers/code-chunker

  - Hosted playground (free with account): https://cloud.chonkie.ai
Happy Chonking!

Ask HN: How realistic is an LLM-powered worm that spreads from PC to PC?

2•anon115•2m ago•0 comments

Classified U.S. Intelligence System Is Down for Now, U.S. Officials Say

https://www.wsj.com/politics/national-security/classified-u-s-intelligence-system-is-down-for-now...
1•uxhacker•2m ago•0 comments

Self-Improvement Memory Enables Continual Recursive Self-Improvement

https://arxiv.org/abs/2611.18856
2•omarsar•5m ago•0 comments

Decision model for all inputs "videos, voices, txt "

https://www.rightnowai.co/rightnow
2•OsamaJaber•5m ago•0 comments

Meet Your Neighbors

https://meetyourneighbors.ai/
1•ot•6m ago•0 comments

CVE-2026-114938 – Java4Shell

https://www.cve.org/CVERecord?id=CVE-2026-114938
3•dark_days•6m ago•0 comments

Possible theory of why things are breaking

4•freecodeio•7m ago•1 comments

Linux sandboxes from S3. 93-99% cheaper than always on for AI Agents.

https://github.com/dualeai/exec-sandbox
1•clemlesne•9m ago•0 comments

Tesla needs wireless charging fast, or its $30,000 Cybercab can’t charge at home

https://electrek.co/2026/11/17/tesla-cybercab-home-charging-wireless-problem/
2•cisc•10m ago•0 comments

RNZ's election results analysis reveals a complicated, moody electorate

https://www.rnz.co.nz/news/politics_election-2026/2646498/rnz-s-election-results-analysis-reveals...
1•colinprince•11m ago•0 comments

Human.md

https://www.human-md.ai/
3•ray__•12m ago•0 comments

How many agents can you supervise and be productive?

https://www.ascendaflow.com/benchmark
1•marisolchallen•12m ago•1 comments

A physical archive of your digital life

https://commonlogue.com/
1•dchi04•13m ago•0 comments

The AI Alignment Meeting

https://folderal.substack.com/p/the-ai-alignment-meeting
1•jeffLowery•14m ago•0 comments

Gemini 4 Carbon cracked an unread 1466 Sforza cipher

https://www.fofr.ai/bianco-1466-cipher
2•underdeserver•14m ago•0 comments

Stripe Press

https://stripe.press/
1•Luc•15m ago•0 comments

As Voyager 1 reaches one light-day from Earth, scientists reflect on its past

https://www.livescience.com/space/space-exploration/out-there-moving-into-the-unknown-as-voyager-...
2•speckx•15m ago•0 comments

Lighting That Feels Just Right

https://blog.maximeheckel.com/posts/lighting-that-feels-just-right/
1•squidhunter•15m ago•0 comments

Isomorphic Labs $5B Series C

https://www.cnbc.com/2026/11/17/isomorphic-labs-series-c-ai-drug-discovery-.html
3•kaycebasques•17m ago•0 comments

The Rise and Fall of Syquest

https://dfarq.homeip.net/the-rise-and-fall-of-syquest/
1•nickt•18m ago•0 comments

We read 5,692 CVE exploit repos. Six in ten were real. 66 were bait

https://flawharbor.com/blog/harbor-check-poc-repos.html
1•symbi•18m ago•0 comments

prtop – htop for your merge requests to wake Claude Code

https://tomycesaille.github.io/prtop/
1•jorislacance•20m ago•0 comments

Computer Graphics from Scratch

https://www.gabrielgambetta.com/computer-graphics-from-scratch/index.html
3•ibobev•21m ago•1 comments

Don't Buy a Tesla for "Full Self-Driving"

https://onefoottsunami.com/2026/11/17/dont-buy-a-tesla-for-full-self-driving/
1•speckx•22m ago•0 comments

Preliminary Post Incident Report for Microsoft 365

https://dreamstation.systems/miscStuff/20261115PreliminaryPostIncidentReportForMicrosoft365.docx
1•robinpie•22m ago•0 comments

Show HN: I've asked AI to generate portraits per country

https://arnov.github.io/faces/index.html
1•arno_v•24m ago•0 comments

The short age of dragon riders

https://atilloy.com/2026/11/16/the-short-age-of-dragon-riders/
1•mathgenius•25m ago•0 comments

Why software compatibility fixes sometimes fail: the diagnosis was wrong

https://nhapplied.co.uk/doctor
2•NHApplied•26m ago•0 comments

SHA-256 collision at 39 steps for ~98 bit found

https://twitter.com/privacymage/status/2120852111089127833/photo/1
4•binyu•27m ago•0 comments

Show HN: I wrote a ProtectServer FM in Zig and skipped the vendor compiler

https://808bits.com/articles/thales-protectserver-fm-zig/
2•NiklasKoehler•27m ago•0 comments