frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Show HN: Fast and Quality Code Chunking with Chonkie

1•snyy•1y ago
Hi HN,

We’re Chonkie (https://github.com/chonkie-inc/chonkie) — we build open source tools that help split documents into meaningful chunks for use with AI models.

When you use LLMs over large documents or codebases, you often need to break them into smaller parts to fit the model’s context window. Our chunkers do this in a smart way: they preserve structure and meaning, so only the most relevant pieces are passed into the model. This reduces hallucinations, avoids confusion, and improves performance and accuracy.

Today we’re launching our Code Chunker — a fast, structure-aware way to break down source code into high-quality, token-aware chunks.

How it works:

(See the code: https://github.com/chonkie-inc/chonkie/blob/main/src/chonkie...)

Code Chunker uses tree-sitter (https://tree-sitter.github.io/tree-sitter/) to parse your code into an abstract syntax tree (AST). It then recursively merges and groups nodes in a way that respects both code structure and token limits.

It supports all languages that tree-sitter supports, and is designed to preserve formatting and semantics. Large functions or class definitions won’t be split in the middle of a block — instead, we dive recursively into the AST to produce clean, coherent chunks that fit your configured token budget.

What it’s useful for:

  - Embedding-based code search

  - RAG (retrieval-augmented generation) over codebases

  - Long-context analysis of code

  - Preparing repos for fine-tuning or pretraining
Try it out:

  - Open source package: https://docs.chonkie.ai/chunkers/code-chunker

  - Hosted playground (free with account): https://cloud.chonkie.ai
Happy Chonking!

Amazon's split personality: AWS embraces bot commerce, main site enforces bans

https://www.theregister.com/ai-and-ml/2026/11/20/amazons-split-personality-aws-embraces-bot-comme...
1•nephihaha•3m ago•0 comments

GTA6 sales hit $3.8bn on first day of release

https://www.theguardian.com/games/2026/nov/21/a-new-record-for-entertainment-grand-theft-auto-vi-...
1•kefabean•5m ago•0 comments

irreplaceable

1•vladten•7m ago•1 comments

Show HN: ClawVet – Offline AST scanner and rug-pull diff for AI agent skills

https://clawvet.vercel.app/
2•bazmou•11m ago•0 comments

Algonquin Dam on Fox River has been hacked

https://www.cbsnews.com/chicago/news/algonquin-dam-fox-river-cyberattack/
3•pretext•15m ago•0 comments

Generative Engine Optimisation: Good Geo Relies on Good SEO

https://bitilia.com/blog/generative-engine-optimisation/
1•bitilia•18m ago•1 comments

We let our AI save memories by itself. What it got wrong in the first two days

https://nexupira.com/news/memory-that-saves-itself
1•jojo960527•19m ago•0 comments

AWS Common Runtime (CRT) v1.0.0 in C

https://aws.amazon.com/blogs/developer/announcing-aws-common-runtime-crt-v1-0-0/
1•begoon•19m ago•0 comments

Three breaches, endless leaks: Your AI supply chain needs canaries

https://www.dragos.com/blog/ai-supply-chain-security-canary-tokens
1•qznc•25m ago•0 comments

Show HN: Factory worker maps AI safety arguments and open questions

https://huggingface.co/datasets/AegisSolis/aegis-solis-crosswalk-rc9-public-research
1•aegissolis•26m ago•0 comments

Microtonal BitcoinCash Transaction Sonifier and Visualizer

https://spiritsofbitcoin.cash/
1•jlmoriart•26m ago•1 comments

Show HN: Jane Street's Socratic Tutor Demo

https://try-now.pages.dev/socratic-tutor/
1•david_2107•29m ago•0 comments

Show HN: PocketTransfer – iPhone to PC/Mac file transfer in the browser

https://pockettransfer.app/
1•wentalau•29m ago•0 comments

Super-hot rock geothermal could end data center energy woes

https://techcrunch.com/2026/11/20/super-hot-rock-geothermal-could-end-data-center-energy-woes/
1•Brajeshwar•29m ago•0 comments

Tracing the Emergence of Extreme Ultraviolet Lithography (CSET, 2024) [pdf]

https://cset.georgetown.edu/wp-content/uploads/CSET-Tracing-the-Emergence-of-Extreme-Ultraviolet-...
1•0sake_rs•29m ago•1 comments

Anthropic's IPO raises questions about AI safety and shareholder pressure

https://apnews.com/article/anthropic-history-openai-dario-amodei-ipo-53fb21bd250b936d93c5c8885af8...
4•ComputedCamp995•35m ago•1 comments

Fokus, a local macOS site blocker and screen time tracker with an MCP server

https://github.com/Jnooo-cloud/fokus
1•Janooo•36m ago•0 comments

Spacecraft reaches Mercury on second-ever mission to enter its orbit

https://www.theguardian.com/science/2026/nov/20/spacecraft-mercury-second-ever-mission-orbit-bepi...
1•geden•38m ago•0 comments

SleepyTube – make YouTube sound and look sleepier (Chrome extension)

https://github.com/kandekar007/sleepytube
1•adi-kandekar•38m ago•1 comments

Removing binder.c

https://lwn.net/Articles/1102982/
1•signa11•41m ago•0 comments

More Suicides at Spy, Cyber Agencies Scrutinized by Congress

https://www.bloomberg.com/news/articles/2026-11-10/more-suicides-at-spy-cyber-agencies-scrutinize...
3•_tk_•49m ago•1 comments

Show HN: Learnmap, learn programming in a playful way

https://learnmap.sh
1•erikpau•53m ago•0 comments

GameShark, cheat devices, and PlayStation runtime function patching

https://generalreasoning.com/blog/2026/11/19/gameshark-patching.html
1•skimdesk•53m ago•0 comments

Dragons and Serpents in Sussex

http://www.sussexarch.org.uk/saaf/dragon.html
1•zeristor•56m ago•0 comments

Maestri – Infinite canvas for orchestrating agents

https://www.themaestri.app/en
3•surrTurr•56m ago•1 comments

Trump vows to prevent US companies from building data centres abroad

https://www.cnbc.com/2026/11/20/trump-data-centers-ai-companies-tax.html
1•niemandhier•57m ago•2 comments

Notion – A New Shape for Work

https://www.notion.com/a-new-shape-for-work
1•simianwords•59m ago•0 comments

Proof of the Riemann Hypothesis [01/01/2026]

https://robertedwardgrant.com/proof-of-the-riemann-hypothesis-a-geometric-constraint-on-prime-flu...
1•fred_the_chair•1h ago•0 comments

Gemini training on chats opt-out is finally here

https://support.google.com/gemini/answer/13594961?hl=en
2•adinafxv•1h ago•1 comments

What's the difference between an integer and a pointer?

https://blog.regehr.org/archives/1621
1•germanrabbit•1h ago•0 comments