frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Show HN: Fast and Quality Code Chunking with Chonkie

1•snyy•1y ago
Hi HN,

We’re Chonkie (https://github.com/chonkie-inc/chonkie) — we build open source tools that help split documents into meaningful chunks for use with AI models.

When you use LLMs over large documents or codebases, you often need to break them into smaller parts to fit the model’s context window. Our chunkers do this in a smart way: they preserve structure and meaning, so only the most relevant pieces are passed into the model. This reduces hallucinations, avoids confusion, and improves performance and accuracy.

Today we’re launching our Code Chunker — a fast, structure-aware way to break down source code into high-quality, token-aware chunks.

How it works:

(See the code: https://github.com/chonkie-inc/chonkie/blob/main/src/chonkie...)

Code Chunker uses tree-sitter (https://tree-sitter.github.io/tree-sitter/) to parse your code into an abstract syntax tree (AST). It then recursively merges and groups nodes in a way that respects both code structure and token limits.

It supports all languages that tree-sitter supports, and is designed to preserve formatting and semantics. Large functions or class definitions won’t be split in the middle of a block — instead, we dive recursively into the AST to produce clean, coherent chunks that fit your configured token budget.

What it’s useful for:

  - Embedding-based code search

  - RAG (retrieval-augmented generation) over codebases

  - Long-context analysis of code

  - Preparing repos for fine-tuning or pretraining
Try it out:

  - Open source package: https://docs.chonkie.ai/chunkers/code-chunker

  - Hosted playground (free with account): https://cloud.chonkie.ai
Happy Chonking!

De-fanging dangerous triggers in GitHub Actions across an organization

https://pyfound.blogspot.com/2026/11/de-fanging-dangerous-triggers-in-github.html
1•lumpa•3m ago•0 comments

Show HN: Kruftle – reclaim disk space with each toolchain's own clean command

https://kruftle.dizitart.com/
1•anidotnet•3m ago•0 comments

Air Force Maj. Jason Watson has been in solitary confinement for almost 100 days

https://www.stripes.com/branches/air_force/2026-11-17/jason-watson-pretrial-hearing-uscmj-charges...
1•o_-_o•5m ago•1 comments

Nvidia investigating Gentlemen ransomware breach claims

https://www.bleepingcomputer.com/news/security/nvidia-investigating-gentlemen-ransomware-breach-c...
1•saikatsg•7m ago•0 comments

AgentCall – Bring your agents to multilingual online meetings using skill

https://github.com/pattern-ai-labs/agentcall/
1•jomon003•8m ago•1 comments

Wrytix

https://wrytix.com/
1•Keywordstat•14m ago•0 comments

Always Today: a to-do app where the AI does the legwork rows

https://alwaystoday.uk/
1•guzhikov•15m ago•0 comments

Lithium-free batteries of the future: research at Politecnico di Milano

https://www.polimi.it/en/research/news/detail/lithium-free-batteries-future-research-politecnico-...
1•znpy•15m ago•0 comments

Nvidia stops shipping US-approved AI chips to China

https://www.reuters.com/world/china/nvidia-stops-shipping-us-approved-ai-chips-china-2026-11-18/
1•nelaggy•15m ago•0 comments

AI security engineering: the same failures, but new ways to attack

https://zimmergren.net/ai-security-engineering/
1•zimmergren•15m ago•0 comments

Inference After Coding Agents

https://arxiv.org/abs/2611.23174
2•doener•15m ago•0 comments

The PSF end-of-year fundraiser

https://simonwillison.net/2026/Nov/18/psf-end-of-year-fundraiser/
1•lumpa•18m ago•0 comments

Copilot is "the new OS for work," and now Office runs inside of it

https://arstechnica.com/gadgets/2026/11/microsoft-calls-copilot-the-new-os-for-work-at-ignite-key...
1•nobody9999•20m ago•0 comments

AWS CVEs 2024-2026: Is AI Finding the Bugs, or Writing Them?

https://blog.technodrone.cloud/2026/11/aws-cves-ai-finding-or-writing-bugs.html
1•maishsk•21m ago•0 comments

Who's Delivering GTA 6? A Ripe Atlas Measurement Story

https://labs.ripe.net/author/emileaben/whos-delivering-gta-6-a-ripe-atlas-measurement-story/
3•minusf•21m ago•0 comments

Fusion power is about to start working

https://www.economist.com/the-world-ahead/2026/11/18/fusion-power-is-about-to-start-working-says-...
1•mpweiher•22m ago•0 comments

180B Parameters, 32GB of RAM: Pushing Moe Inference to the Limit

https://huggingface.co/blog/moe-expert-streaming
1•joshcsimmons•22m ago•0 comments

EU Commission targeted in cyberattack, Politico reports

https://www.reuters.com/legal/litigation/eu-commission-targeted-cyberattack-politico-reports-2026...
1•gvieri•27m ago•0 comments

The deployment path is a product

https://jgothelf.substack.com/p/the-deployment-path-is-a-product
1•flail•29m ago•0 comments

The Hunt for the Death Valley Germans

https://otherhand.org/home-page/search-and-rescue/the-hunt-for-the-death-valley-germans/
2•oumua_don17•34m ago•0 comments

Hip Hop HQ Magazine

https://www.thisishiphophq.com.ng/#/
1•mrb3njamins•37m ago•0 comments

Educational Demo of the Solar system and its visitors

https://compo.tv/w/perihelion
1•mfbilling•37m ago•0 comments

Show HN: My CTF Setup with BlackArch in Docker

https://coding-overhead.com/posts/blackarch-docker/
1•ismd•40m ago•0 comments

Game theory's cure for corruption (2015)

https://aeon.co/essays/game-theory-s-cure-for-corruption-make-us-all-cops
1•sammatilda•43m ago•0 comments

EUVD-2024-5409 Synology and Beestation – Improper Encoding or Escaping of Output

https://euvd.enisa.europa.eu/vulnerability/EUVD-2024-54098
1•austinallegro•44m ago•0 comments

The New Mind Control – The internet has spawned subtle forms of influence (2016)

https://aeon.co/essays/how-the-internet-flips-elections-and-alters-our-thoughts
1•sammatilda•44m ago•0 comments

DeepMind Employees Are Unionising

https://www.gdmunion.co.uk
3•nervai•46m ago•0 comments

Where do we go from here?

https://nicolasbouliane.com/blog/life-after-ai
1•nicbou•47m ago•0 comments

How to Create a Virtual Background Using the Red5 WebRTC SDK

https://www.red5.net/blog/virtual-backgrounds-using-the-red5-webrtc-sdk/
1•mondainx•48m ago•0 comments

What's New in PHP 8.6

https://www.php.net/releases/8.6/en.php
32•netol•48m ago•11 comments