frontpage.
newsnewestaskshowjobs

Open Source @Github

fp.

Open in hackernews

An Enterprise-Level Retrieval-Augmented Generation System

https://comfyai.app/article/llm-applications/enterprise-level-rag-hands-on-practice-II
6•zljdanceholic•1y ago

Comments

zljdanceholic•1y ago
How can we search the wanted key information from 10,000+ pages of PDFs within 2.5 hours? For fact check, how do we implement it so that answers are backed by page-level references, minimizing hallucinations?

RAG-Challenge-2 is a great open-source project by Ilya Rice that ranked 1st at the Enterprise RAG Challenge, which has 4500+ lines of code for implementing a high-performing RAG system. It might seem overwhelming to newcomers who are just beginning to learn this technology. Therefore, to help you get started quickly—and to motivate myself to learn its ins and outs—I’ve created a complete tutorial on this.

We have a complete graph to explain its workflow where multiple tools are used: Docling for parsing PDFs, LangChain for chunking text, faiss for vectorization and similarity searching, and chatgpt for LLMs.

Besides, I also outline the codeflow, demonstrating the running logic involving multiple python files where starters can easily get lost. Different files are colored differently. The purpose of showing this is not letting you memorize all of these file relationships. It works better for you to check the source code yourself and use this as a reference if you find yourself lost in the code.

The original project of Ilya Rice design its RAG system for answering questions of annual reports from companies, so he only designed three types of question response format for that challenge: a name, a number, or a boolean. But to ask questions about technical stuff, we absolutely ask general questions like How does RoPE work? to know about some concepts and the like Therefore, I further modify the system logic to fit this need by customizing an AnswerWithRAGContextExplanationPrompt class and automatically matching the most related chapter and corresponding pages via searching through all faiss databases (only retrieve the top-1) The final performance is demonstrated below (not cherry-picked, only tested once).

- What's the steps to train a nanoGPT from scratch? The answer is:

Training nanoGPT from scratch involves several clearly defined steps. First, set up the environment by installing necessary libraries, using either Anaconda or Google Colab, and then download the dataset (e.g., tinyShakespeare). Next, tokenize the text into numerical representations and split the data into training and validation sets. Define the model architecture including token/positional embeddings, transformer blocks with multi-head self-attention and feed-forward networks, and layer normalization. Configure training hyperparameters and set up an optimizer (such as AdamW). Proceed with a training loop that performs forward passes, computes loss, backpropagates, and updates parameters, while periodically evaluating performance on both training and validation data. Finally, use the trained model to generate new text from a given context.

All code are provided on Colab and the tutorial is referenced here. Hope this helps!

Putting Task Expertise into RL Achieves SOTA Performance on Text-to-SQL

https://thinkingmachines.ai/news/putting-task-expertise-into-rl/
1•gmays•7s ago•0 comments

We're going to compete for $90B market. Looking for a technical cofounder

https://www.screenomics.co/
1•boisovamir•2m ago•1 comments

Building HBOM for a Router

https://worklifenotes.com/2026/11/13/building-hbom-for-a-router/
1•taleodor•4m ago•0 comments

Show HN: Perfect Numbers Generator – p = (2P and 1)/3

https://www.academia.edu/resource/work/181008238
2•A19dammer91•7m ago•0 comments

Show HN: Word Gambits, Daily Puzzle Capture Chess Pieces to Form Words

https://wordgambits.com/
1•hamsal•13m ago•0 comments

Show HN: Redacted - Stop Handing Claude Your Secrets

https://github.com/svn-arv/redacted/
1•svn-arv•13m ago•0 comments

Mapping the Jensen Huang Saga

https://www.patreon.com/MaxRead/posts/mapping-jensen-175257766
2•themgt•15m ago•0 comments

Reading speculations concerning the first ultraintelligent machine

https://www.jefftk.com/p/reading-speculations-concerning-the-first-ultraintelligent-machine
1•dmd•16m ago•0 comments

Brain Implants Give ALS Patients the Voice to Speak and Sing

https://www.nytimes.com/2026/11/14/science/brain-implants-als-voice.html
1•garyrob•17m ago•0 comments

AI and the City

https://www.stripeeconomics.com/p/ai-and-the-city
1•gmays•19m ago•0 comments

A hard rain's a-gonna fall: parsing JSON in Rust

https://bitfieldconsulting.com/posts/hard-rain-json-rust
1•teleforce•22m ago•0 comments

TuringFonts: Publish Uncrawable and Uncopiable Texts

https://jfmdev.github.io/TuringFonts/index.html
1•negura•23m ago•0 comments

Qwen4-Max

https://qwen.ai/blog?id=qwen4-max
11•ilreb•27m ago•2 comments

Race Against the (Mac)Hine: Local Priv Esc in Microsoft Defender for macOS

https://spaceraccoon.dev/race-against-the-machine-defender-macos-lpe/
1•chiroyce•28m ago•0 comments

Mega Speed Says Alice Huang Left Company in August 2025

https://www.nytimes.com/2026/11/15/business/megaspeed-alice-huang-nvidia-china.html
2•firesteelrain•31m ago•0 comments

Apple's Smart Glasses Strategy Hinges on Siri and iPhone Apps

https://www.macrumors.com/2026/11/15/apple-smart-glasses-hinge-on-siri/
3•mgh2•34m ago•0 comments

Universal XSS in uBlock Origin from website-driven element picker

https://iter.ca/post/ubo-uxss/
23•zaik•37m ago•2 comments

We Only Wanted an Answer – How Two Worlds Save Themselves – Short Story

https://medium.com/@mkraft_berlin/we-only-wanted-an-answer-how-two-worlds-save-themselves-and-two...
2•mkraft_berlin•42m ago•1 comments

Show HN: AquarIum – pick your favorite (AI-generated) fish tank

https://galleries.artfish.ai/aquarium/
4•yenniejun111•45m ago•0 comments

Will the AI boom continue? Forecasting the trajectory of the AI industry

https://forecastingresearch.substack.com/p/will-the-ai-boom-continue-forecasting
3•gmays•47m ago•0 comments

A.I. Stories in the Age of AI Hype: A Conversation with Ted Chiang

https://buttondown.com/charliejane/archive/ai-stories-in-the-age-of-ai-hype-a-conversation/
4•jdkee•49m ago•0 comments

Woolworths customers report funds stolen from digital gift cards

https://www.abc.net.au/news/2026-11-16/woolworths-customers-report-money-stolen-from-digital-gift...
4•Gaishan•50m ago•0 comments

From Vision to Veyron: The Evolution of a Bugatti Icon

https://newsroom.bugatti.com/en/press-releases/from-vision-to-veyron
1•melenaboija•54m ago•0 comments

Apple launches research study to explore the use of AI for health

https://www.apple.com/newsroom/2026/11/apple-launches-research-study-to-explore-the-use-of-ai-for...
2•johnbehnke•58m ago•0 comments

Show HN: Agency – give your agents the computer you own

https://agency.makors.xyz/
4•bryce2010va•1h ago•0 comments

TRMI: Multiplayer ADE with Shared Memory

https://trmi.dev/download
1•slyda99•1h ago•1 comments

How NewsNinja syncs all its data in less than 1 MB of iCloud

https://ainews.ninja/blog/icloud-key-value-store-sync/
1•xeo84•1h ago•0 comments

"Who is Hiring?" postings stuck near 12-year lows as AI reaches half of all jobs

https://www.hntrends.com/2026/november.html
50•myth_drannon•1h ago•18 comments

Visualizing Metra (2011)

https://www.c82.net/blog/visualizing-metra
1•tomeraberbach•1h ago•0 comments

Agentic AI Designs an Analog Synthesizer

https://cpldcpu.github.io/2026/11/14/touchsynth/
1•oidar•1h ago•1 comments