Google has added a brand new crawler to its checklist of Google Crawlers and consumer brokers, this one is called GoogleOther. It’s described as a “generic crawler that could be utilized by varied product groups for fetching publicly accessible content material from websites.”
For instance, it could be used for one-off crawls for inner analysis and improvement, Google defined. The GoogleOther crawler at all times obeys robots.txt guidelines for its consumer agent token and the worldwide consumer agent (*), and makes use of the identical IP ranges as Googlebot.
The Person agent token is “GoogleOther” and the complete consumer agent string is “GoogleOther.”
Gary Illyes from Google defined on LinkedIn that this new crawler will “substitute a few of Googlebot’s different jobs like R&D crawls to unencumber some crawl capability for Googlebot.”
Gary wrote, “We added a brand new crawler, GoogleOther to our checklist of crawlers that finally will take some pressure off of Googlebot. This can be a no-op change for you, but it surely’s fascinating nonetheless I reckon. As we optimize how and what Googlebot crawls, one factor we needed to make sure is that Googlebot’s crawl jobs are solely used internally for constructing the index that is utilized by Search. For this we added a brand new crawler, GoogleOther, that may substitute a few of Googlebot’s different jobs like R&D crawls to unencumber some crawl capability for Googlebot. The brand new crawler makes use of the identical infrastructure as Googlebot and so it has the identical limitations and options as Googlebot: hostload limitations, robotstxt (although totally different consumer agent token), http protocol model, fetch measurement, you title it. It is mainly Googlebot underneath a special title.”
There is no such thing as a touch upon if this crawler might or is probably not used for Google Bard.
So in case you see this new crawler, GoogleOther, that is what it’s about.
I feel they need to have named it “MiniMeGoogleBot.”
Only a clarification from Gary:
> utilized by Google product groups for internally constructing the Google index
the opposite approach round: Googlebot is used for index constructing, GoogleOther for the opposite duties traditionally owned by Googlebot
— Gary 鯨理／경리 Illyes (@methode) April 20, 2023
like, random one-off crawls we used to do for studying one thing. Wanna study what number of websites *require* an settle for language request header? run a crawl job. This was with googlebot, now it is gonna be with GoogleOther.
Please do not overthink it, it is actually that boring.— Gary 鯨理／경리 Illyes (@methode) April 20, 2023
Discussion board dialogue at LinkedIn.
