List of Google's special-case crawlers

The special-case crawlers are used by specific Google products where there's an agreement between the crawled site and the product about the crawl process. For example, AdsBot ignores the global robots.txt user agent (*) with the ad publisher's permission. The general technical properties of Google's crawlers also apply to the special-case crawlers.

The special-case crawlers may ignore robots.txt rules and so they operate from a different IP range than the common crawlers. The IP ranges are published in the special-crawlers.json object. The special-case crawlers' reverse DNS mask matches rate-limited-proxy-***-***-***-***.google.com.

The following list shows the special-case crawlers, their user agent strings as they appear in the HTTP requests, their user agent tokens for the User-agent: line in robots.txt, and the products that are affected by crawl preferences for the crawler. The list is not exhaustive, it only covers the requestors that are more likely to show up in log files and that we've received questions about.

APIs-Google

User-Agent in HTTP requests	APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)
robots.txt	User-agent token in robots.txt APIs-Google The global user agent (*) is ignored. Example robots.txt group user-agent: APIs-Google allow: /archive/1Q84 disallow: /archive/
Affected products	Crawling preferences addressed to the APIs-Google user agent affect the delivery of push notification messages by Google APIs.

AdsBot Mobile Web

User-Agent in HTTP requests	Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
robots.txt	User-agent token in robots.txt AdsBot-Google-Mobile The global user agent (*) is ignored. Example robots.txt group user-agent: AdsBot-Google-Mobile allow: /archive/1Q84 disallow: /archive/
Affected products	Crawling preferences addressed to the AdsBot-Google-Mobile user agent affect Google Ads' ability to check web page ad quality.

AdsBot

User-Agent in HTTP requests	AdsBot-Google (+http://www.google.com/adsbot.html)
robots.txt	User-agent token in robots.txt AdsBot-Google The global user agent (*) is ignored. Example robots.txt group user-agent: AdsBot-Google allow: /archive/1Q84 disallow: /archive/
Affected products	Crawling preferences addressed to the AdsBot-Google user agent affect Google Ads' ability to check web page ad quality.

AdSense

User-Agent in HTTP requests	Desktop agent Mediapartners-Google Mobile agent (Various mobile device types) (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
robots.txt	User-agent token in robots.txt Mediapartners-Google The global user agent (*) is ignored. Example robots.txt group user-agent: Mediapartners-Google allow: /archive/1Q84 disallow: /archive/
Affected products	Crawling preferences addressed to the Mediapartners-Google user agent affect Google AdSense. The AdSense crawler visits participating sites in order to provide them with relevant ads.

Google-Safety

User-Agent in HTTP requests	Google-Safety
robots.txt	The Google-Safety user agent ignores robots.txt rules.
Affected products	The Google-Safety user agent handles abuse-specific crawling, such as malware discovery for publicly posted links on Google properties. As such it's unaffected by crawling preferences.