Original Summary

Most robots.txt tools still treat AI as a single checkbox. That stopped being true a while ago, and it's quietly costing people visibility. OpenAI, Anthropic and Google have all split their tokens by purpose. GPTBot is training. OAI-SearchBot is what puts you in ChatGPT's search results. ChatGPT-User fetches a page because a person just asked about it. Blocking GPTBot keeps you out of model training and does nothing to your ChatGPT search visibility. Blocking OAI-SearchBot does the exact opposite. Most people want one of those and end up with the other, usually by copying a snippet from an article written before the split happened. The worst version is the inverted pair: search bot blocked, training bot allowed. You're invisible in the results and your content still feeds the model. Nothing warns you about it. So I built a tester that checks a path against all 16 publicly documented tokens at once instead of one at a time, shows which line made the decision and at what line number, and flags the inverted pairs. Precedence follows RFC 9309, which is its own trap: the longest matching pattern wins regardless of where it sits in the file, an Allow only beats a Disallow on an exact length tie, and a crawler matched by name ignores the wildcard group entirely. Every token came from the operator's own documentation rather than a listicle, because this is the corner of the subject where secondhand lists rot fastest. Runs entirely in the browser, nothing gets uploaded, no signup. https://utilix.app/tools/robots-txt-tester Happy to be told I got a token wrong, that's the failure mode I care about most.   submitted by   /u/UtilixApp [link]   [comments]

  • 情报分类：技术学习与提效
  • 分类依据：内容涉及技术、AI、软件工具或工程实践
  • 信息来源：Reddit · SideProject
  • 发布时间：2026/9/17 05:48:37