frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

Show HN: Fast and Quality Code Chunking with Chonkie

1•snyy•1y ago
Hi HN,

We’re Chonkie (https://github.com/chonkie-inc/chonkie) — we build open source tools that help split documents into meaningful chunks for use with AI models.

When you use LLMs over large documents or codebases, you often need to break them into smaller parts to fit the model’s context window. Our chunkers do this in a smart way: they preserve structure and meaning, so only the most relevant pieces are passed into the model. This reduces hallucinations, avoids confusion, and improves performance and accuracy.

Today we’re launching our Code Chunker — a fast, structure-aware way to break down source code into high-quality, token-aware chunks.

How it works:

(See the code: https://github.com/chonkie-inc/chonkie/blob/main/src/chonkie...)

Code Chunker uses tree-sitter (https://tree-sitter.github.io/tree-sitter/) to parse your code into an abstract syntax tree (AST). It then recursively merges and groups nodes in a way that respects both code structure and token limits.

It supports all languages that tree-sitter supports, and is designed to preserve formatting and semantics. Large functions or class definitions won’t be split in the middle of a block — instead, we dive recursively into the AST to produce clean, coherent chunks that fit your configured token budget.

What it’s useful for:

  - Embedding-based code search

  - RAG (retrieval-augmented generation) over codebases

  - Long-context analysis of code

  - Preparing repos for fine-tuning or pretraining
Try it out:

  - Open source package: https://docs.chonkie.ai/chunkers/code-chunker

  - Hosted playground (free with account): https://cloud.chonkie.ai
Happy Chonking!

Listening to music as an active process

https://www.autodidacts.io/listening-to-music-as-an-active-process/
1•Curiositry•3m ago•0 comments

Show HN: Memeo – Turn static photos into lip-synced AI meme videos in seconds

https://memeo.app/
1•shuangge•4m ago•0 comments

New Mexico wanted free childcare for all; what's worked – and what hasn't

https://text.npr.org/nx-s1-6040194
3•1659447091•10m ago•0 comments

Megaspeed says Alice Huang left in 2025

https://apnews.com/article/singapore-megaspeed-alice-huang-nvidia-probe-e9e2e18169dce0b30a34f3971...
1•Jarlakxen•13m ago•0 comments

Recent progress in mathematics, discussed by research mathematicians

https://mathhelm.org/
2•qnleigh•17m ago•0 comments

Mapping cyber risk in the era of AI: Which industries are most exposed?

https://www.moodys.com/web/en/us/insights/data-stories/moodys-2026-cyber-risk-exposure-heat-map-r...
1•gmays•17m ago•0 comments

Zeitpyramide

https://en.wikipedia.org/wiki/Zeitpyramide
1•cornfieldlabs•20m ago•0 comments

Bill Gates Has Earned Billions Investing in Landfills

https://www.propublica.org/article/bill-gates-republic-landfill-methane-pollution
4•Gaishan•21m ago•0 comments

Tell HN: HN is being scraped for emails to scam

3•eks391•29m ago•0 comments

China's Economic Slowdown Deepens

https://www.scmp.com/economy/economic-indicators/article/3374531/chinas-economic-activity-falters...
2•lavendermon•29m ago•0 comments

Guineas' population was much larger than expected

https://www.reddit.com/r/overpopulation/comments/1ricesi/guineas_population_was_much_larger_than_...
1•oliculipolicula•30m ago•0 comments

Extreme Tux Racer for Trimui v0.1

https://github.com/dimitrovs/extremetuxracer-sdl/releases/tag/trimui-v0.1
1•elwebmaster•32m ago•0 comments

Show HN: HappyWan – A browser-based prototype for text-to-video creation

https://happywan.pro
2•rollingcarollin•34m ago•2 comments

Putting Task Expertise into RL Achieves SOTA Performance on Text-to-SQL

https://thinkingmachines.ai/news/putting-task-expertise-into-rl/
2•gmays•40m ago•0 comments

We're going to compete for $90B market. Looking for a technical cofounder

https://www.screenomics.co/
2•boisovamir•42m ago•1 comments

Building HBOM for a Router

https://worklifenotes.com/2026/11/13/building-hbom-for-a-router/
1•taleodor•44m ago•0 comments

Show HN: Perfect Numbers Generator – p = (2P and 1)/3

https://www.academia.edu/resource/work/181008238
2•A19dammer91•47m ago•0 comments

Show HN: Word Gambits, Daily Puzzle Capture Chess Pieces to Form Words

https://wordgambits.com/
1•hamsal•53m ago•0 comments

Show HN: Redacted - Stop Handing Claude Your Secrets

https://github.com/svn-arv/redacted/
1•svn-arv•53m ago•0 comments

Mapping the Jensen Huang Saga

https://www.patreon.com/MaxRead/posts/mapping-jensen-175257766
2•themgt•55m ago•0 comments

Reading speculations concerning the first ultraintelligent machine

https://www.jefftk.com/p/reading-speculations-concerning-the-first-ultraintelligent-machine
1•dmd•56m ago•0 comments

Brain Implants Give ALS Patients the Voice to Speak and Sing

https://www.nytimes.com/2026/11/14/science/brain-implants-als-voice.html
1•garyrob•57m ago•0 comments

AI and the City

https://www.stripeeconomics.com/p/ai-and-the-city
1•gmays•59m ago•0 comments

A hard rain's a-gonna fall: parsing JSON in Rust

https://bitfieldconsulting.com/posts/hard-rain-json-rust
1•teleforce•1h ago•0 comments

TuringFonts: Publish Uncrawable and Uncopiable Texts

https://jfmdev.github.io/TuringFonts/index.html
1•negura•1h ago•0 comments

Qwen4-Max

https://qwen.ai/blog?id=qwen4-max
88•ilreb•1h ago•13 comments

Race Against the (Mac)Hine: Local Priv Esc in Microsoft Defender for macOS

https://spaceraccoon.dev/race-against-the-machine-defender-macos-lpe/
1•chiroyce•1h ago•0 comments

Mega Speed Says Alice Huang Left Company in August 2025

https://www.nytimes.com/2026/11/15/business/megaspeed-alice-huang-nvidia-china.html
4•firesteelrain•1h ago•0 comments

Apple's Smart Glasses Strategy Hinges on Siri and iPhone Apps

https://www.macrumors.com/2026/11/15/apple-smart-glasses-hinge-on-siri/
4•mgh2•1h ago•0 comments

Universal XSS in uBlock Origin from website-driven element picker

https://iter.ca/post/ubo-uxss/
73•zaik•1h ago•10 comments