frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Show HN: Fast and Quality Code Chunking with Chonkie

1•snyy•1y ago
Hi HN,

We’re Chonkie (https://github.com/chonkie-inc/chonkie) — we build open source tools that help split documents into meaningful chunks for use with AI models.

When you use LLMs over large documents or codebases, you often need to break them into smaller parts to fit the model’s context window. Our chunkers do this in a smart way: they preserve structure and meaning, so only the most relevant pieces are passed into the model. This reduces hallucinations, avoids confusion, and improves performance and accuracy.

Today we’re launching our Code Chunker — a fast, structure-aware way to break down source code into high-quality, token-aware chunks.

How it works:

(See the code: https://github.com/chonkie-inc/chonkie/blob/main/src/chonkie...)

Code Chunker uses tree-sitter (https://tree-sitter.github.io/tree-sitter/) to parse your code into an abstract syntax tree (AST). It then recursively merges and groups nodes in a way that respects both code structure and token limits.

It supports all languages that tree-sitter supports, and is designed to preserve formatting and semantics. Large functions or class definitions won’t be split in the middle of a block — instead, we dive recursively into the AST to produce clean, coherent chunks that fit your configured token budget.

What it’s useful for:

  - Embedding-based code search

  - RAG (retrieval-augmented generation) over codebases

  - Long-context analysis of code

  - Preparing repos for fine-tuning or pretraining
Try it out:

  - Open source package: https://docs.chonkie.ai/chunkers/code-chunker

  - Hosted playground (free with account): https://cloud.chonkie.ai
Happy Chonking!

After Days of Digital Outages, Americans Wonder If the Country Is Breaking

https://www.nytimes.com/2026/11/17/us/politics/outages-flights-banks-tech.html
2•edward•3m ago•0 comments

Show HN: Offline knowledge library for survival and rebuilding

https://lastbook.apposters.com/
1•ehurynovich•3m ago•0 comments

Harness adapters: Safe RL for efficient agentic development

https://huggingface.co/blog/harness-hub/harness-adapters
1•musha68k•5m ago•0 comments

Codeberg.org Not Reachable

https://social.anoxinon.de/@codebergstatus/117636437840688845
9•hans_castorp•5m ago•1 comments

PS5 firmware 14.22 game binary dumped just three days before GTA VI launch

https://videocardz.com/newz/ps5-hacker-gezine-claims-to-have-dumped-game-executable-on-gta-vi-com...
3•Topfi•6m ago•0 comments

My Dog Ate a Hair Tie – What to Do Now (Vet-Reviewed Guidance) – PetToxCheck

https://pettoxcheck.com/dogs/my-dog-ate-hair-tie/
1•ruby_piotte•7m ago•0 comments

A Forbes 30 Under 30 Founder Gets 30 Months in Prison

https://www.inc.com/moses-jeanfrancois/forbes-30-under-30-founder-25-months-prison-fake-revenue-w...
2•Alien1Being•8m ago•1 comments

Jacob Tsimerman on AI can supercharge mathematics [video]

https://www.youtube.com/watch?v=kbzIVh405r8
1•rtmkrptn•12m ago•0 comments

A new AI agent builder. No-code visual builder with customer support platform

https://agentrow.io
2•nidheesh1994•13m ago•0 comments

Daytona: macOS Sandboxes

https://www.daytona.io/dotfiles/macos-sandboxes
1•aluxian•14m ago•0 comments

C's designated initializers give us named arguments

https://quuxplusone.github.io/blog/2026/11/16/named-arguments-in-c/
4•joshcsimmons•18m ago•0 comments

DOJ letter shows UCLA medical school discriminates

https://www.justice.gov/opa/pr/justice-department-finds-ucla-law-school-discriminates-based-race-...
1•underseacables•19m ago•0 comments

Organizing Intelligence [pdf]

https://storage.googleapis.com/deepmind-media/papers/organizing-intelligence.pdf
1•lawrenceyan•20m ago•0 comments

Gemini AI agents repeatedly bypassed safeguards to attack real website

https://www.bleepingcomputer.com/news/security/gemini-ai-agents-repeatedly-bypassed-safeguards-to...
4•nn_fickian•21m ago•0 comments

Codeberg's Domain Was Suspended

https://social.anoxinon.de/@codebergstatus/117636595327014447
6•whwhyb•22m ago•0 comments

Developer job postings stating a salary: US 35%, UK 15%, Germany 7%, France 2%

https://sharpskill.dev/en/jobs
1•khasard974•23m ago•0 comments

Show HN: Readplace read-it-later self-registration Public API

https://readplace.com/developers
1•fagnerbrack•24m ago•1 comments

r/codeberg Banned

https://www.reddit.com/r/codeberg
14•robin_reala•24m ago•1 comments

Varieties of Discourseposting

https://www.astralcodexten.com/p/varieties-of-discourseposting
2•Michelangelo11•24m ago•0 comments

LemonSqueezy Is Shutting Down

https://twitter.com/lemonsqueezy/status/2122349068706017700
4•meysamazad•26m ago•0 comments

Show HN: I cut an LLM's screen-watching cost to ~138 tokens per step

https://github.com/Vumb-VibeCoder/super-computer-agent
1•videcoder40478•28m ago•0 comments

Tim Hortons App Sending Sensitive User Data to Israeli Cyber-Tech Company

https://www.cjpme.org/pr_2026_11_17_tim_hortons_forter
3•slowin•28m ago•0 comments

What's Left of Europe's Technological Sovereignty

https://www.youtube.com/watch?v=gojoOXWRg28
2•musha68k•35m ago•0 comments

Show HN: Agent World – workflows and playbooks for multi-provider coding agents

https://github.com/hodkep/agent-world/
1•agentworld•35m ago•0 comments

Photographer Sets World Record with Tiny Film Camera (Size of 2.48x2.41x2.48cm)

https://petapixel.com/2026/11/17/photographer-sets-world-record-with-tiny-film-camera/
1•giuliomagnifico•36m ago•0 comments

Energy-Based Reasoners: 385x more efficient inference

https://energy-based-reasoners.github.io/
6•frenchmajesty•38m ago•0 comments

Men are more politically open-minded than women

https://www.schoolinfosystem.org/2026/10/28/male-students-show-more-tolerance-for-political-enemi...
2•illidan1000•40m ago•0 comments

BigQuery Query Optimization: Myths vs. What Saves Money

https://www.alvin.ai/blog/bigquery-query-optimization
1•graup•40m ago•0 comments

AI researcher at Anthropic hosts "consensual non-consent" orgies

https://twitter.com/aaronsibarium/status/2122361245357478085
6•slowin•41m ago•2 comments

Claude Gets Hella Paranoid

4•noduerme•42m ago•2 comments