List of Google user-triggered fetchers
User-triggered fetchers are initiated by users to perform a fetching function within a Google product. For example, Google Site Verifier acts on a user's request, or a site hosted on Google Cloud (GCP) has a feature that allows the site's users to retrieve an external RSS feed. Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules. The general technical properties of Google's crawlers also apply to the user-triggered fetchers.
The IP ranges the user-triggered fetchers use are
published in the
user-triggered-fetchers.json and
user-triggered-fetchers-google.json
objects. The user-triggered fetchers' reverse DNS mask, depending on whether the fetcher is Google
or user owned, matches ***-***-***-***.gae.googleusercontent.com
or
google-proxy-***-***-***-***.google.com
respectively.
The following list shows the user-triggered fetchers, their user agent strings as they appear in the HTTP requests, and the products they are associated with. The list is not exhaustive, it only covers the requestors that are more likely to show up in log files and that we've received questions about.
Feedfetcher
-
User-Agent in HTTP requests FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)
Associated products Feedfetcher is used for crawling RSS or Atom feeds for Google News and PubSubHubbub. Google Publisher Center
-
User-Agent in HTTP requests GoogleProducer; (+https://developers.google.com/search/docs/crawling-indexing/google-producer)
Associated products Google Publisher Center fetches and processes feeds that publishers explicitly supplied for use in Google News landing pages. Google Read Aloud
-
User-Agent in HTTP requests Mobile agent Mozilla/5.0 (Linux; Android 7.0; SM-G930V Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36 (compatible; Google-Read-Aloud; +https://support.google.com/webmasters/answer/1061943)
Desktop agent Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36
Former agent (deprecated) google-speakr
Associated products Upon user request, Google Read Aloud fetches and reads out web pages using text-to-speech (TTS). Google Site Verifier
-
User-Agent in HTTP requests Mozilla/5.0 (compatible; Google-Site-Verification/1.0)
Associated products Google Site Verifier fetches Search Console verification tokens.