Original Summary

I opened my server log on a Sunday morning expecting to check Googlebot, and Googlebot was fifth. Above it, in order: ChatGPT's search crawler, Bing, Apple, and a name I had to look up. I'd spent the whole summer optimizing for the engine at the bottom of the list. One site, one day, yesterday. OAI-SearchBot, the crawler ChatGPT Search uses to build its index: 1,101 pages. Bingbot, which feeds Copilot and a few AI answer tools: 204. Applebot, for Siri: 84. Googlebot: 62. ChatGPT-User, which is a real person opening one of your links inside a ChatGPT answer: 45. PerplexityBot: 1. Claude's search crawler: 3. Your numbers will be different. That's the point. You should know yours, and nobody sells them, because they're sitting in a file you already have. If you have server logs (nginx, apache, or the log export on Vercel, Cloudflare and Netlify), run this on yesterday's file: grep -c "OAI-SearchBot" access.log grep -c "ChatGPT-User" access.log grep -c "PerplexityBot" access.log grep -c "Claude-SearchBot" access.log grep -c "Googlebot" access.log If you don't have logs, Cloudflare's free analytics has a bot filter that names these crawlers, and Vercel's does under "Bots". Same five names. Two things to look at beyond the counts. Which pages they fetch: if OAI-SearchBot only ever hits your homepage, it knows your name, not your product. And the status codes next to robots.txt: mine had returned a 502 to crawlers 114 times in two weeks, once per app restart, and a 5xx on robots.txt tells every engine to stop crawling the whole site until it can read the file again. I don't know how much of my later recovery that fix caused, but it mattered more than anything I did to the content that month. One rule for reading the result. A zero for PerplexityBot means you are not in Perplexity's retrieval set, full stop. No AEO tool showing you "visible in Perplexity" outranks a zero in your own log. Run it and post your five counts. I want to know if ChatGPT-over-Google is my site or the year.   submitted by   /u/mrkeyoor [link]   [comments]

中文概览

中文标题： ChatGPT的搜索爬虫昨天读了我1101个页面，Google只读了62个

作者称查看服务器日志发现，昨日OAI-SearchBot抓取1101页，Bingbot 204页，Applebot 84页，Googlebot仅62页，ChatGPT-User 45次，PerplexityBot 1次，Claude搜索爬虫3次。他建议用自己已有的nginx、apache或Vercel、Cloudflare、Netlify日志做同样统计，还提醒留意各爬虫抓取哪些页面，以及robots.txt返回的状态码（他曾出现502）；并称PerplexityBot为零就表示你不在其检索集合中。

  • 情报分类：商业与市场研究
  • 分类依据：AI爬虫日志与搜索可见性分析，属商业与市场研究。
  • 信息来源：Reddit · SaaS
  • 发布时间：2026/9/22 01:41:33