- SignalDesk4 hr ago
Original Summary
Hi, I'm trying to get much better at using search engines for technical research. If you just want the question, jump straight to <i>QUESTION</i> below. The rest is context for why I'm asking.<p>---<p>LLMs solve part of the problem today as it helps better information foraging quickly, for instance when what I need is a clear definition. But I have the feeling that there is still much more I could get out of web search if I knew how to use it properly. Serendipity is one obvious example. But I keep running into a problem that makes the whole process very inefficient: context explosion.<p>Take Apache Hadoop.<p>Let's say I encounter the term somewhere and I don't know what it is.<p>I search for it, land on Wikipedia, and get something like:<p>"Apache Hadoop is a collection of open-source software utilities for reliable, scalable, distributed computing. It provides a software framework for distributed storage and processing of big data using the MapReduce programming model."<p>Now the problem is that if I don’t already know the field, this definition creates several new questions and so on.<p>What exactly is MapReduce? What does "programming model" mean in this particular context? "Programming model" is the kind of expression where I understand both words, but that doesn't necessarily tell me precisely what the author means by putting them together here. Then I end up with a lot of tabs open.<p>I'm deliberately using a slightly crude example. Obviously I can keep reading the Wikipedia page. There is an Overview section, HDFS and YARN get introduced, the architecture becomes more concrete, and after a while I can probably build a decent picture of what Hadoop is.<p>So my point isn't that Wikipedia gives a bad definition, or that the information isn't there. It's more that there is a difference between eventually reconstructing the answer from all of that and getting something like: "Hadoop is basically a system for turning a cluster of machines into distributed storage plus distributed compute for large datasets."<p>When I asked ChatGPT about Hadoop, it could give me something at roughly that level immediately. Maybe the answer here is simply that an LLM is better for that particular job. I don't know and don't care. I'm more interested in is whether I'm missing something on the web-search side.<p>---<p>However I'm not completely new to web search either. I know that different kinds of sources are useful for different things. I know about forums, aggregators, documentation, Stack Overflow, GitHub issues, Hacker News and so on. I also know enough of the technical web ecosystem that a SERP isn't completely flat to me anymore: I recognize names like USENIX, CNCF, LWN, good company engineering blogs, vendor documentation, independent technical blogs, and so on, and I often already have some sense of which results are worth opening and which ones raise a red flag. I also know that useful searches are often indirect, and that one good result can lead to a good author, another site, another term, another discussion, etc.<p>And yet I keep running into the same problem: context explosion. So even though I feel like I already have quite a few of the pieces, I don't feel that I have a precise method for this.<p>I've also had the opposite experience, where web search was much better than asking ChatGPT. Recently I was looking into some FastAPI questions around how to organize things — when something should be a query, when it should be an operation, how people actually structure that boundary. I ended up finding Stack Overflow discussions that made the issue much clearer to me than the answers I was getting from ChatGPT.<p>So there is obviously a larger question here about how to combine both efficiently, but that isn’t really the question I’m trying to ask in this post, its much narrower. Here it is:<p>---
- 情报分类:工作与职业机会
- 分类依据:内容涉及招聘、求职或职业发展
- 信息来源:Hacker News 新项目
- 发布时间:2026/9/27 00:26:43
- No replies yet