How HTTP status codes, and network and DNS errors affect Google Search
This page describes how different
HTTP status codes,
network errors, and DNS errors affect Google Search. We cover the top 20 status codes that
Googlebot
encountered on the web, and the most prominent network and DNS errors. More exotic status
codes, such as
418 (I'm a teapot)
,
aren't covered. All issues mentioned on this page generate a corresponding error or warning in
Search Console's
Crawl Stats report.
HTTP status codes
HTTP status codes are generated by the server that's hosting the site when it responds to a request made by a client, for example a browser or a crawler. Every HTTP status code has a different meaning, but often the outcome of the request is the same. For example, there are multiple status codes that signal redirection, but their outcome is the same.
Search Console generates error messages for status codes in the 4xx–5xx
range,
and for failed redirections (3xx
). If the server responded with a
2xx
status code, the content received in the response may be considered for
indexing.
HTTP status codes | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
2xx (success) |
Google considers the content for indexing. If the content suggests an error, for example an empty page or an error message, Search Console will show a soft 404 error.
|
|||||||||||
3xx (redirection) |
Googlebot follows up to 10 redirect hops. If the crawler doesn't receive content within 10 hops, Search Console will show a redirect error in the site's Index Coverage report. The number of hops Googlebot follows is user agent dependent; for example, Googlebot Smartphone may have a different value than Googlebot Image. Any content Googlebot received from the redirecting URL is ignored, and the final target URL's content is considered for indexing.
|
|||||||||||
4xx (client errors) |
Google's indexing pipeline doesn't consider URLs that return a
Any content Googlebot received from URLs that return a
|
|||||||||||
5xx (server errors) |
Any content Googlebot received from URLs that return a
|
Network and DNS errors
Network and DNS errors have quick, negative effects on a URL's presence in Google Search.
Googlebot treats network timeouts, connection reset, and DNS errors similarly to
5xx
server errors. In case of network errors, crawling immediately starts
slowing down, as a network error is a sign that the server may not be able to handle the
serving load. Since Googlebot couldn't reach the server hosting the site, it also hasn't
received any content from the server. The lack of content means that Google can't index the
crawled URLs, and already indexed URLs that are unreachable will be removed from Google's
index within days. Search Console may generate errors for each respective error.
Debug network errors
These errors happen before Google starts crawling a URL or while Google is crawling the URL. Since the errors may occur before the server can respond and so there's no status code that can hint at issues, diagnosing these errors can be more challenging. To debug timeout and connection reset errors:
- Look at your firewall settings and logs. There may be an overly-broad blocking rule set. Make sure that Googlebot IP addresses are not blocked by any firewall rule.
- Look at the network traffic. Use tools like tcpdump and Wireshark to capture and analyze TCP packets, and look for anomalies that point to a specific network component or server module.
- If you can't find anything suspicious, contact your hosting company.
The error may be in any server component that handles network traffic. For example, overloaded
network interfaces may drop packets leading to timeouts (inability to establish a connection)
and reset connections (RST
packet sent because a port was mistakenly closed).
Debug DNS errors
DNS errors are most commonly caused by misconfiguration, but they may be also caused by a firewall rule that's blocking Googlebot DNS queries. To debug DNS errors, do the following:
-
Inspect your firewall rules. Make sure that
none of Google's IPs
are blocked by any firewall rule, and that both
UDP
andTCP
requests are allowed. -
Look at your DNS records. Double check that your
A
andCNAME
records are pointing to the right IP addresses and hostname, respectively. For example:dig +nocmd example.com a +noall +answer
dig +nocmd www.example.com cname +noall +answer
-
Check that all your name servers are pointing to the correct IP addresses of your
site. For example:
dig +nocmd example.com ns +noall +answer
example.com. 86400 IN NS a.iana-servers.net. example.com. 86400 IN NS b.iana-servers.net.dig +nocmd @a.iana-servers.net example.com +noall +answer
example.com. 86400 IN A 93.184.216.34dig +nocmd @b.iana-servers.net example.com +noall +answer
... - If you've made changes to your DNS configuration within the last 72 hours, you may need to wait for your changes to propagate across the global DNS network.
- If you're running your own DNS server, make sure it's healthy and that it's not overloaded.