frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Show HN: Fast and Quality Code Chunking with Chonkie

1•snyy•1y ago
Hi HN,

We’re Chonkie (https://github.com/chonkie-inc/chonkie) — we build open source tools that help split documents into meaningful chunks for use with AI models.

When you use LLMs over large documents or codebases, you often need to break them into smaller parts to fit the model’s context window. Our chunkers do this in a smart way: they preserve structure and meaning, so only the most relevant pieces are passed into the model. This reduces hallucinations, avoids confusion, and improves performance and accuracy.

Today we’re launching our Code Chunker — a fast, structure-aware way to break down source code into high-quality, token-aware chunks.

How it works:

(See the code: https://github.com/chonkie-inc/chonkie/blob/main/src/chonkie...)

Code Chunker uses tree-sitter (https://tree-sitter.github.io/tree-sitter/) to parse your code into an abstract syntax tree (AST). It then recursively merges and groups nodes in a way that respects both code structure and token limits.

It supports all languages that tree-sitter supports, and is designed to preserve formatting and semantics. Large functions or class definitions won’t be split in the middle of a block — instead, we dive recursively into the AST to produce clean, coherent chunks that fit your configured token budget.

What it’s useful for:

  - Embedding-based code search

  - RAG (retrieval-augmented generation) over codebases

  - Long-context analysis of code

  - Preparing repos for fine-tuning or pretraining
Try it out:

  - Open source package: https://docs.chonkie.ai/chunkers/code-chunker

  - Hosted playground (free with account): https://cloud.chonkie.ai
Happy Chonking!

Wolf at the Door: Apple Rethinking Mac Permissions in the Age of AI

https://macanorak.com/a-wolf-at-the-door-apple-is-rethinking-mac-permissions-for-the-age-of-ai-ag...
1•barbariangrunge•52s ago•0 comments

Five Eligible CPUs, a Four-CPU Quota: What Runtimes Report and What It Costs

https://octri.dev/blog/cpu-affinity-vs-cgroup-quota
1•khalidkhair•3m ago•0 comments

Intuit to sell Mailchimp at about a third of its 2021 purchase price

https://www.reuters.com/legal/transactional/intuit-sell-mailchimp-bending-spoons-368-billion-2026...
1•gostsamo•3m ago•2 comments

Stock perps show asymmetric profitability outside traditional market hours

https://blog.aperiodic.io/p/closed-market-stock-perps
1•szemy2•3m ago•0 comments

Show HN: Bocalito – A Meal Prep Planner

https://bocalito.com
1•eskombro•4m ago•0 comments

The Brain–is wider than the Sky

https://blue-continuum.com/the-brain-is-wider-than-the-sky
1•dnetesn•4m ago•0 comments

Waymo stalls in high San Francisco floodwaters during storm

https://ktla.com/news/california/waymo-stalls-san-francisco-floodwaters/
1•mikhael•6m ago•0 comments

Unit-Checker for Excel

https://github.com/psteckler/daubee
1•psteckler•7m ago•0 comments

A New Era of Deterministic AI Begins: OLMapp Emerges from Stealth

https://olmapp.ai/index.html
1•northlondoner•10m ago•1 comments

The fix can't wait. What breaks when it lands?

https://irlabs.ai/blog/the-fix-cant-wait/
2•marssaxman•12m ago•1 comments

Batteries: The Invention That Untethered the World

https://stepchangeshow.substack.com/p/batteries-the-invention-that-untethered
1•toomuchtodo•12m ago•1 comments

P&G Aims to Reduce Manufacturing Workforce Up to 30% by 2030

https://www.bloomberg.com/news/articles/2026-11-19/p-g-aims-to-reduce-manufacturing-workforce-up-...
1•petethomas•13m ago•0 comments

Celliad – a daily Excel game (French)

https://celliad.com/
1•didir98•14m ago•0 comments

An FBI agent called a Claude user about his chats

https://austinpatrick.substack.com/p/when-should-ai-labs-alert-police
3•AustinLikesAI•15m ago•0 comments

Last remaining NIH-owned chimpanzees will retire to a sanctuary next year

https://www.science.org/content/article/last-remaining-nih-owned-chimpanzees-will-retire-sanctuar...
1•gmays•19m ago•0 comments

Show HN: 15MB transcription model that runs in the browser/device

https://starbright.quickthoughts.ca/
2•quickthoughts•19m ago•1 comments

Galleri – Screen for 50 cancers with a single blood test

https://www.galleri.com/
1•numlocked•20m ago•0 comments

The trouble with serendipit: scientific discoveries

https://www.worksinprogress.news/p/the-trouble-with-serendipity
1•greazy•22m ago•0 comments

From 8-bit Amiga to Synthwave: Original tracks composed on an Amiga in the 90s

https://vosmeer.com/
1•doener•24m ago•0 comments

RegExr: Temporarily Offline

https://regexr.com/
5•somebudyelse•24m ago•7 comments

The [uncertain] Four Seasons: Vivaldi re-composed for 2050 climate projections

https://the-uncertain-four-seasons.info/
1•__fst__•25m ago•0 comments

Visa, Mastercard confirm cyberattacks that may have exposed customers' info

https://www.theregister.com/security/2026/11/19/chase-visa-mastercard-confirm-third-party-cyberat...
3•Betelbuddy•25m ago•1 comments

Intuit to Sell Mailchimp to Bending Spoons for $3.68B

https://www.cnbc.com/2026/11/19/intuit-mailchimp-sale-bending-spoons.html
2•jsk2600•27m ago•1 comments

C-to-Rust Translation Validation at the LLVM IR Level via AI-Assisted Proof

https://arxiv.org/abs/2611.18036
1•matt_d•29m ago•0 comments

AMD dives on short-seller report of alleged China exposure

https://www.cnbc.com/2026/11/19/amd-falls-on-short-seller-report-of-alleged-china-exposure.html
2•Betelbuddy•30m ago•0 comments

A Novel AI-Native GPU Kernel Programming Language for Autonomous Kernel Tuning

https://al.radbox.org/doi/10.1145/3828523.3857041
1•matt_d•30m ago•0 comments

Register your app on open-source repositories

https://developer.android.com/developer-verification/guides/open-source-app-registration
3•exploraz•31m ago•0 comments

IPO: Anthropic Bankruptsy Likelier Than Success [video][15m]

https://www.youtube.com/watch?v=a5pHmGq1mEs
4•Bender•33m ago•0 comments

Neoism – Coolest Agent Framework Around Town

https://neoism.ai/
1•Mr_Settle•33m ago•0 comments

Math Books by Robert Ghrist

https://www2.math.upenn.edu/~ghrist/books/
2•ibobev•35m ago•1 comments