frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Show HN: Fast and Quality Code Chunking with Chonkie

1•snyy•1y ago
Hi HN,

We’re Chonkie (https://github.com/chonkie-inc/chonkie) — we build open source tools that help split documents into meaningful chunks for use with AI models.

When you use LLMs over large documents or codebases, you often need to break them into smaller parts to fit the model’s context window. Our chunkers do this in a smart way: they preserve structure and meaning, so only the most relevant pieces are passed into the model. This reduces hallucinations, avoids confusion, and improves performance and accuracy.

Today we’re launching our Code Chunker — a fast, structure-aware way to break down source code into high-quality, token-aware chunks.

How it works:

(See the code: https://github.com/chonkie-inc/chonkie/blob/main/src/chonkie...)

Code Chunker uses tree-sitter (https://tree-sitter.github.io/tree-sitter/) to parse your code into an abstract syntax tree (AST). It then recursively merges and groups nodes in a way that respects both code structure and token limits.

It supports all languages that tree-sitter supports, and is designed to preserve formatting and semantics. Large functions or class definitions won’t be split in the middle of a block — instead, we dive recursively into the AST to produce clean, coherent chunks that fit your configured token budget.

What it’s useful for:

  - Embedding-based code search

  - RAG (retrieval-augmented generation) over codebases

  - Long-context analysis of code

  - Preparing repos for fine-tuning or pretraining
Try it out:

  - Open source package: https://docs.chonkie.ai/chunkers/code-chunker

  - Hosted playground (free with account): https://cloud.chonkie.ai
Happy Chonking!

A New Shape for Work

https://www.notion.com/a-new-shape-for-work
1•pikdum•2m ago•0 comments

Making torch_remat work in TorchTitan

https://docs.pytorch.org/devlogs/distributed/2026-11-19-torch-remat-in-torchtitan/
1•matt_d•6m ago•0 comments

Forensic dissection of Roland J-6's 64 presets: measuring the hidden settings

https://github.com/liotier/RolandStuff/blob/main/J-6_presets_measurement/README.md
1•liotier•10m ago•0 comments

Is Ssense Pivoting to AI Slop?

https://www.thecut.com/article/is-ssense-pivoting-to-ai-slop.html
1•randycupertino•12m ago•1 comments

Your target market isn't demographic (2025)

https://longform.asmartbear.com/target-market/
2•mooreds•12m ago•0 comments

How to generate an accessible color palette from a single color

https://taisho.design/en/blog/accessible-color-palette-from-one-color
2•PabloDiFIlippo•15m ago•0 comments

Scaling Postgres to 1M durable workflows per second

https://www.dbos.dev/blog/one-million-workflows
3•KraftyOne•15m ago•0 comments

Show HN: Primeward – Exploring a stateful recurrence for the next prime

1•alexhr•23m ago•0 comments

Vulcan Street Plant

https://en.wikipedia.org/wiki/Vulcan_Street_Plant
1•_doctor_love•24m ago•0 comments

Catching Crumbs from the Table [pdf]

https://gwern.net/doc/fiction/science-fiction/2000-chiang.pdf
2•gchamonlive•25m ago•1 comments

Show HN: Jam – A music making game where you play gigs to earn studio gear

https://jam.resonantdsp.com
1•garrinm•27m ago•0 comments

Athanor (furnace)

https://en.wikipedia.org/wiki/Athanor
1•_doctor_love•33m ago•0 comments

Show HN: Park – save and recall work context across repos and machines

https://svandragt.github.io/park/
1•pacifika•35m ago•3 comments

Ask HN: How do you keep code review skills when agents write most of the code?

1•robinmin•35m ago•0 comments

Re: Riemann

https://borretti.me/article/re-riemann
6•_dain_•35m ago•0 comments

Petrucci Music Library

https://imslp.org
1•yetanotheruser•42m ago•0 comments

Help github just deleted me

https://nate.mecca1.net/posts/2026-11-20_help/
39•01nate•42m ago•19 comments

AI models change their answers when they are monitored

https://www.jordannews.jo/Section-129/Technology/AI-models-change-their-answers-when-they-are-mon...
2•nreece•44m ago•0 comments

Show HN: SaturnSQL Context, telling AI tools what the database means

https://saturnsql.com/context-layer
2•infr88•46m ago•0 comments

Real-Time Video on Embedded Linux Cameras with WebRTC

https://interrupt.memfault.com/blog/real-time-video-webrtc
2•hasheddan•46m ago•0 comments

Uncontacted Indigenous people emerge from Amazon forest to ask for food

https://www.washingtonpost.com/world/2026/11/19/uncontacted-indigenous-people-emerge-amazon-fores...
12•Teever•47m ago•0 comments

A human liver has been kept alive outside the body for a record seven weeks

https://www.technologyreview.com/2026/11/19/1148934/a-human-liver-has-been-kept-alive-outside-the...
9•bookofjoe•48m ago•1 comments

Officials approved the sale of 580k Nvidia chips to Chinese companies

https://www.washingtonpost.com/technology/2026/11/20/officials-approved-sale-580000-nvidia-chips-...
7•cdrnsf•48m ago•3 comments

Serious cyberattack thought backed by wealthy operators

https://www.ruv.is/english/2026-11-20-serious-cyberattack-thought-backed-by-wealthy-operators-493411
4•Bender•49m ago•1 comments

White House removes cyber-security chief after series of tech outages

https://www.bbc.com/news/articles/cwz0k3nn5p5mo
4•Lyngbakr•52m ago•0 comments

How to Be a 10x Engineer in 2027 and Beyond

https://serce.me/posts/2026-11-19-how-to-become-a-10x-engineer-in-2027-and-beyond
2•SerCe•53m ago•0 comments

ICE details Princeton neuroscientist claiming asylum

https://www.theguardian.com/us-news/2026/nov/20/princeton-doctoral-student-reportedly-detained-ice
7•angry_octet•54m ago•2 comments

The Riemann Hypothesis and a Happy Meal

https://zlxie.com/2026/11/21/riemann-hypothesis/
3•mostbill•56m ago•0 comments

Sir Brian May adds new dimension to Hera asteroid

https://www.esa.int/articles/about-us/directorates/operations/space-safety/hera/sir-brian-may-add...
16•mooreds•58m ago•0 comments

Professional Counting Solutions for Your Industry

https://countthings.com/
2•mooreds•59m ago•0 comments