frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Show HN: Fast and Quality Code Chunking with Chonkie

1•snyy•1y ago
Hi HN,

We’re Chonkie (https://github.com/chonkie-inc/chonkie) — we build open source tools that help split documents into meaningful chunks for use with AI models.

When you use LLMs over large documents or codebases, you often need to break them into smaller parts to fit the model’s context window. Our chunkers do this in a smart way: they preserve structure and meaning, so only the most relevant pieces are passed into the model. This reduces hallucinations, avoids confusion, and improves performance and accuracy.

Today we’re launching our Code Chunker — a fast, structure-aware way to break down source code into high-quality, token-aware chunks.

How it works:

(See the code: https://github.com/chonkie-inc/chonkie/blob/main/src/chonkie...)

Code Chunker uses tree-sitter (https://tree-sitter.github.io/tree-sitter/) to parse your code into an abstract syntax tree (AST). It then recursively merges and groups nodes in a way that respects both code structure and token limits.

It supports all languages that tree-sitter supports, and is designed to preserve formatting and semantics. Large functions or class definitions won’t be split in the middle of a block — instead, we dive recursively into the AST to produce clean, coherent chunks that fit your configured token budget.

What it’s useful for:

  - Embedding-based code search

  - RAG (retrieval-augmented generation) over codebases

  - Long-context analysis of code

  - Preparing repos for fine-tuning or pretraining
Try it out:

  - Open source package: https://docs.chonkie.ai/chunkers/code-chunker

  - Hosted playground (free with account): https://cloud.chonkie.ai
Happy Chonking!

On the probability of robots becoming more creative than humans

https://www.thetedkarchive.com/library/ted-kaczynski-on-the-probability-of-robots-becoming-more-c...
1•maplet•1m ago•0 comments

X's silent update yesterday broke third-party scraper pipelines

https://twitterapi.io/blog/x-searchtimeline-nov-2026-breaking-change-user-legacy-removed
1•jt_park•3m ago•0 comments

Ask HN: Are you prepared to rent intelligence from AI providers?

1•piratesAndSons•6m ago•0 comments

MaldocTrace – emulating malicious Office/OneNote docs to see what they'd run

https://github.com/debugactiveprocess/MaldocTrace
1•ironbits•9m ago•0 comments

The oddball who founded UPS and changed the world

https://www.economist.com/culture/2026/11/19/the-oddball-who-founded-ups-and-changed-the-world
1•andsoitis•10m ago•0 comments

California Research Measure Is a Windfall for a Billionaire's Pet Project

https://www.nytimes.com/2026/11/18/science/california-prop-38-gary-michelson.html
1•lxm•13m ago•0 comments

Ask HN: Printing documents in the age of AI cyberattacks

2•andsoitis•13m ago•1 comments

Show HN: GitHub Plays Linux

https://github.com/seg6
2•seg6•17m ago•0 comments

Informal Verification

https://www.nikolasgoebel.com/2026/11/22/informal-verification.html
2•ComNik•20m ago•0 comments

A teen gamer and his mother helped rescue a missing girl

https://www.bbc.com/news/articles/cmq8lyv8lkjwo
2•1659447091•21m ago•0 comments

Two Hands in a Local Box – SQLite state gating and negative provenance for LLMs

https://github.com/HenryOscarArthurMcDonald/two-hands-local-box
2•HenryMcDonald•22m ago•0 comments

Can AI Make Its Own Manufacturing Decisions?

https://semiengineering.com/can-ai-make-its-own-manufacturing-decisions/
2•gmays•25m ago•0 comments

Current Efforts Aren't Enough to Slow the AI Cyber Virus

https://www.bridgewater.com/research-and-insights/current-efforts-arent-enough-to-slow-the-ai-cyb...
2•DenisM•26m ago•1 comments

Show HN: System 1.5 Model – Inkling-2-Mini Decision

https://huggingface.co/eltokh7/inkling-2-mini-decision
3•eltokh7•27m ago•0 comments

Learning the Language of Machine Learning

https://news.mit.edu/2026/learning-language-of-machine-learning-jacob-andreas-1117
2•colinprince•27m ago•0 comments

Hardening NixOS After I Got Hacked

https://lantian.pub/en/article/modify-computer/hardening-nixos-after-i-got-hacked.lantian/
3•Shank•30m ago•0 comments

Upgrade my old MacBook turned home server

https://andreagao.com/posts/upgrade-macbook-turned-home-server/
3•freediver•34m ago•0 comments

Fast and Incremental iPhone Backups for Windows

https://mahdis24.tech/iphone-backup-sync/
2•mahdi_s24•36m ago•0 comments

Show HN: Hikari.ImGui, a pure C# re-implementation of Dear ImGui

https://github.com/LeastActionTech/Hikari.ImGui
2•charm137•37m ago•0 comments

One Month, 130 New Listings On donatemonero.org

https://monerocoalition.org/one-month-130-new-listings-on-donatemonero-org/
3•Cider9986•39m ago•0 comments

The Alpöge–Claude proof of the RH explained for the Riemann zeta function [pdf]

https://www.math.uwaterloo.ca/~mrubinst/alpoge_claude_grh/zeta-paper.pdf
2•literallyanythi•39m ago•2 comments

Show HN: A modern take on the Go language

https://github.com/telgatech/gpp
2•u89012•45m ago•0 comments

First Recorded Death "Involving a Rectal Explosion During Self-Gratification"

https://www.vice.com/en/article/man-dies-from-rectal-injuries-after-silly-experiment-with-compres...
3•spankibalt•47m ago•2 comments

Together.ai: Prompt Leakage for Requests

https://www.together.ai/blog/security-event-prompt-leakage-for-requests-in-june-and-july-of-2026
2•Fe2O3•47m ago•0 comments

Since last year, we asked: "What if AI solved the Riemann Hypothesis?"

https://www.youtube.com/watch?v=zqzk43bSdDI
7•droidjj•48m ago•0 comments

Tiling 3D Space by (Translates of) a Single Polycube Is Undecidable

https://edemaine.github.io/translation-tiling-paper/
2•nill0•53m ago•0 comments

US Agriculture Department and FBI hit by cyberattacks, media report

https://www.france24.com/en/americas/20261121-us-agriculture-department-fbi-hit-by-cyberattacks-m...
2•enjoyyourlife•53m ago•0 comments

Relicensing to CC BY-SA and making it visible in Schema metadata

https://shkspr.mobi/blog/2026/11/relicensing-to-cc-by-sa-and-making-it-visible-in-schema-metadata/
2•ajdude•1h ago•0 comments

The Most Popular Car Colors in America

https://www.iseecars.com/most-popular-car-colors-study
8•cisc•1h ago•1 comments

Show HN: Lifetime Black Friday Deals (Mega List)

https://www.blackfridaydeals.directory/black-friday-lifetime-deals
2•bfdd•1h ago•0 comments