frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Show HN: Fast and Quality Code Chunking with Chonkie

1•snyy•1y ago
Hi HN,

We’re Chonkie (https://github.com/chonkie-inc/chonkie) — we build open source tools that help split documents into meaningful chunks for use with AI models.

When you use LLMs over large documents or codebases, you often need to break them into smaller parts to fit the model’s context window. Our chunkers do this in a smart way: they preserve structure and meaning, so only the most relevant pieces are passed into the model. This reduces hallucinations, avoids confusion, and improves performance and accuracy.

Today we’re launching our Code Chunker — a fast, structure-aware way to break down source code into high-quality, token-aware chunks.

How it works:

(See the code: https://github.com/chonkie-inc/chonkie/blob/main/src/chonkie...)

Code Chunker uses tree-sitter (https://tree-sitter.github.io/tree-sitter/) to parse your code into an abstract syntax tree (AST). It then recursively merges and groups nodes in a way that respects both code structure and token limits.

It supports all languages that tree-sitter supports, and is designed to preserve formatting and semantics. Large functions or class definitions won’t be split in the middle of a block — instead, we dive recursively into the AST to produce clean, coherent chunks that fit your configured token budget.

What it’s useful for:

  - Embedding-based code search

  - RAG (retrieval-augmented generation) over codebases

  - Long-context analysis of code

  - Preparing repos for fine-tuning or pretraining
Try it out:

  - Open source package: https://docs.chonkie.ai/chunkers/code-chunker

  - Hosted playground (free with account): https://cloud.chonkie.ai
Happy Chonking!

Anthropic is not living up to its ideals

https://so8res.substack.com/p/anthropic-is-not-living-up-to-its
1•yurivish•4m ago•1 comments

Levels.fyi for AI Models

https://www.levels.fyi/ai
1•Zaheer•4m ago•0 comments

Tetsuya Chiba Has Died. He Drew Some of the Greatest Manga of All Time

https://www.nytimes.com/2026/11/16/obituaries/tetsuya-chiba-dead.html
1•bookofjoe•5m ago•1 comments

One nuke could wipe out low earth orbit in a week

https://spaceweather.com/archive.php?view=1&day=19&month=11&year=2026
1•MeteorMarc•6m ago•0 comments

Expanding Android developer verification in 2027

https://developer.android.com/blog/posts/android-developer-verification-2027
1•sunaookami•7m ago•0 comments

JPMorgan, Visa, Mastercard say third-party vendors targeted in cyberattack

https://www.reuters.com/legal/government/jpmorgan-says-third-party-vendors-targeted-cyberattack-2...
4•6thbit•7m ago•0 comments

2026: AGI

https://twitter.com/gradypb/status/2011491957730918510
1•pretext•7m ago•0 comments

Device Connect for AWS Device Farm

https://github.com/aws/device-connect-devicefarm
1•aristeiae•7m ago•0 comments

Use Rex to connect to your VMs

https://blog.exe.dev/rex
1•keleftheriou•7m ago•0 comments

Drop Agents.md and Claude.md

https://drmike.xyz/posts/drop-agents-md-and-claude-md/
1•speckx•8m ago•0 comments

Cognitive Republic – the book, essay chapter by essay chapter, is on the way

https://steady.page/en/cognitive-republic/posts/4fdc6a77-b2fa-4ea8-89e5-ed34cc733359
2•smomara•8m ago•0 comments

PostgreSQL 19 Released with More Performance Improvements

https://www.phoronix.com/news/PostgreSQL-19-Released
1•Bender•10m ago•0 comments

Interactive Free Investor Finder Resource Tool for Founders

https://colaeb.com/investors
1•startupfriendly•10m ago•0 comments

Show HN: AdversaryPR – A Claude skill to keep up with AI-written PR's

https://github.com/bugsworthai/adversary
1•curtisspope•10m ago•0 comments

Introducing Model Factory

https://www.primeintellect.ai/blog/introducing-model-factory
1•gmays•10m ago•0 comments

Another critical PHP security release has dropped

https://www.theregister.com/patches/2026/11/18/another-critical-php-security-release-has-dropped/...
2•Bender•11m ago•1 comments

GPT-Live-1.1: Full-duplex voice model for multilingual conversations

https://developers.openai.com/api/docs/models/gpt-live-1.1
2•pretext•11m ago•0 comments

I see my team's work (humans and agents) work now

https://github.com/Irere123/worklog
1•irere123•11m ago•1 comments

How to tell a cyberattack from a plain old outage

https://www.wired.com/story/how-to-tell-if-its-a-cyberattack-or-just-an-outage/
1•Bender•12m ago•0 comments

Artificial Intelligence and the Production of Scientific Knowledge

https://medium.com/@smoking2man/artificial-intelligence-and-the-production-of-scientific-knowledg...
1•Aldus551•15m ago•0 comments

Ask HN: Where do you see Software in 5 years?

2•Odbyteusz•16m ago•1 comments

Show HN: TIE Fighter co-op in 11 days, built from 11 years of Wing Commander

https://fireflyk64.github.io/makingtf/
2•daniel_rh•16m ago•1 comments

An AI agent that pays for every thought and dies when the money runs out

https://www.youtube.com/watch?v=MySG62ItZKU
1•housec84•19m ago•0 comments

The New Era of Wikimedia Traffic: Thinking Beyond Pageviews

https://diff.wikimedia.org/2026/11/19/the-new-era-of-wikimedia-traffic-thinking-beyond-pageviews/
1•cdrnsf•19m ago•0 comments

Show HN: Postdate – share a link now that opens at a time you choose

https://postdate.io/
1•LightMorpheus•20m ago•0 comments

Bram Stoker's Notes for Dracula

https://jillianhess.substack.com/p/re-noted-bram-stokers-notes-for-dracula
1•speckx•21m ago•0 comments

Ask HN: Why no AI-led memory vulns in Python?

1•uticus•23m ago•0 comments

SakaQL: A new general-purpose relational programming language

https://www.saka.io/blog/sakaql-general-purpose-relational-programming-language/
1•eejdoowad•24m ago•1 comments

Musk backs away from MAGA after Republicans' midterm losses

https://apnews.com/article/elon-musk-maga-donald-trump-midterm-elections-b8bb6796cad315517dec4b31...
4•jethronethro•24m ago•1 comments

Rustrak, a self-hosted error tracker for your existing Sentry SDKs

https://github.com/rustrak/rustrak
1•abianS0•28m ago•0 comments