You can verify if a web crawler accessing your server really is Googlebot (or another Google user agent). This is useful if you're concerned that spammers or other troublemakers are accessing your site while claiming to be Googlebot. Google doesn't post a public list of IP addresses for website owners to allowlist. This is because these IP address ranges can change, causing problems for any website owners who have hard-coded them, so you must run a DNS lookup as described next.
Verify that Googlebot is the crawler
Use command line tools
- Run a reverse DNS lookup on the accessing IP address from your logs, using the
- Verify that the domain name is either
- Run a forward DNS lookup on the domain name retrieved in step 1 using the
hostcommand on the retrieved domain name. Verify that it's the same as the original accessing IP address from your logs.
> host 18.104.22.168 22.214.171.124.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com. > host crawl-66-249-66-1.googlebot.com crawl-66-249-66-1.googlebot.com has address 126.96.36.199
> host 188.8.131.52 184.108.40.206.in-addr.arpa domain name pointer rate-limited-proxy-66-249-90-77.google.com. > host rate-limited-proxy-66-249-90-77.google.com rate-limited-proxy-66-249-90-77.google.com has address 220.127.116.11
Use automatic solutions
Google doesn't provide software libraries that can identify its crawlers. You can use an open source library to verify Googlebot.