Hi, I'm trying to get much better at using search engines for technical research. If you just want the question, jump straight to QUESTION below. The rest is context for why I'm asking.

---

LLMs solve part of the problem today as it helps better information foraging quickly, for instance when what I need is a clear definition. But I have the feeling that there is still much more I could get out of web search if I knew how to use it properly. Serendipity is one obvious example. But I keep running into a problem that makes the whole process very inefficient: context explosion.

Take Apache Hadoop.

Let's say I encounter the term somewhere and I don't know what it is.

I search for it, land on Wikipedia, and get something like:

"Apache Hadoop is a collection of open-source software utilities for reliable, scalable, distributed computing. It provides a software framework for distributed storage and processing of big data using the MapReduce programming model."

Now the problem is that if I don’t already know the field, this definition creates several new questions and so on.

What exactly is MapReduce? What does "programming model" mean in this particular context? "Programming model" is the kind of expression where I understand both words, but that doesn't necessarily tell me precisely what the author means by putting them together here. Then I end up with a lot of tabs open.

I'm deliberately using a slightly crude example. Obviously I can keep reading the Wikipedia page. There is an Overview section, HDFS and YARN get introduced, the architecture becomes more concrete, and after a while I can probably build a decent picture of what Hadoop is.

So my point isn't that Wikipedia gives a bad definition, or that the information isn't there. It's more that there is a difference between eventually reconstructing the answer from all of that and getting something like: "Hadoop is basically a system for turning a cluster of machines into distributed storage plus distributed compute for large datasets."

When I asked ChatGPT about Hadoop, it could give me something at roughly that level immediately. Maybe the answer here is simply that an LLM is better for that particular job. I don't know and don't care. I'm more interested in is whether I'm missing something on the web-search side.

---

However I'm not completely new to web search either. I know that different kinds of sources are useful for different things. I know about forums, aggregators, documentation, Stack Overflow, GitHub issues, Hacker News and so on. I also know enough of the technical web ecosystem that a SERP isn't completely flat to me anymore: I recognize names like USENIX, CNCF, LWN, good company engineering blogs, vendor documentation, independent technical blogs, and so on, and I often already have some sense of which results are worth opening and which ones raise a red flag. I also know that useful searches are often indirect, and that one good result can lead to a good author, another site, another term, another discussion, etc.

And yet I keep running into the same problem: context explosion. So even though I feel like I already have quite a few of the pieces, I don't feel that I have a precise method for this.

I've also had the opposite experience, where web search was much better than asking ChatGPT. Recently I was looking into some FastAPI questions around how to organize things — when something should be a query, when it should be an operation, how people actually structure that boundary. I ended up finding Stack Overflow discussions that made the issue much clearer to me than the answers I was getting from ChatGPT.

So there is obviously a larger question here about how to combine both efficiently, but that isn’t really the question I’m trying to ask in this post, its much narrower. Here it is:

---

LibreTube: An alternative front end for YouTube

Comments