Stay organized with collections
Save and categorize content based on your preferences.
Tuesday, August 05, 2008
When Googlebot crawls the web, it often finds what we call an "infinite space". These are very
large numbers of links that usually provide little or no new content for Googlebot to index. If
this happens on your site, crawling those URLs may use unnecessary bandwidth, and could result in
Googlebot failing to completely index the real content on your site.
Recently, we started notifying site owners when we discover this problem on their web sites. Like
most messages we send, you'll find them in
Webmaster Tools
in the Message Center. You'll probably want to know right away if Googlebot has this
problem—or other problems—crawling your sites. So verify your site with Webmaster
Tools, and check the Message Center every now and then.
Examples of an infinite space
The classic example of an "infinite space" is a calendar with a "Next Month" link. It may be
possible to keep following those "Next Month" links forever! Of course, that's not what you want
Googlebot to do. Googlebot is smart enough to figure out some of those on its own, but there are
a lot of ways to create an infinite space and we may not detect all of them.
Another common scenario is websites which provide for filtering a set of search results in many
ways. A shopping site might allow for finding clothing items by filtering on category, price,
color, brand, style, etc. The number of possible combinations of filters can grow exponentially.
This can produce thousands of URLs, all finding some subset of the items sold. This may be
convenient for your users, but is not so helpful for the Googlebot, which just wants to find
everything&mdashl;once!
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],[],[[["\u003cp\u003eGooglebot can encounter "infinite spaces" on websites, which are vast numbers of links with minimal unique content, potentially hindering efficient indexing.\u003c/p\u003e\n"],["\u003cp\u003eWebsite owners are notified of such issues through Google Webmaster Tools, emphasizing the importance of site verification and regular monitoring of the Message Center.\u003c/p\u003e\n"],["\u003cp\u003eCommon examples of "infinite spaces" include endless calendar navigations and excessive filtering options on e-commerce sites.\u003c/p\u003e\n"],["\u003cp\u003eSolutions involve using robots.txt to block dynamic link categories or employing the "nofollow" attribute for specific links, while ensuring all essential content remains accessible to Googlebot.\u003c/p\u003e\n"]]],["Googlebot identifies \"infinite spaces\" on websites, which are vast numbers of links with minimal unique content. These spaces can hinder Googlebot's indexing efficiency. Site owners are notified about this issue in Webmaster Tools' Message Center. Infinite spaces often stem from features like endless calendars or multiple search result filters. Solutions include using `robots.txt` to block dynamically generated links or employing the `nofollow` attribute on problematic links. Webmaster Tools provides further guidance on recognizing and correcting these issues.\n"],null,["# To infinity and beyond? No!\n\nTuesday, August 05, 2008\n\n\nWhen Googlebot crawls the web, it often finds what we call an \"infinite space\". These are very\nlarge numbers of links that usually provide little or no new content for Googlebot to index. If\nthis happens on your site, crawling those URLs may use unnecessary bandwidth, and could result in\nGooglebot failing to completely index the real content on your site.\n\n\nRecently, we started notifying site owners when we discover this problem on their web sites. Like\nmost messages we send, you'll find them in\n[Webmaster Tools](https://search.google.com/search-console)\nin the Message Center. You'll probably want to know right away if Googlebot has this\nproblem---or other problems---crawling your sites. So verify your site with Webmaster\nTools, and check the Message Center every now and then.\n\nExamples of an infinite space\n-----------------------------\n\n\nThe classic example of an \"infinite space\" is a calendar with a \"Next Month\" link. It may be\npossible to keep following those \"Next Month\" links forever! Of course, that's not what you want\nGooglebot to do. Googlebot is smart enough to figure out some of those on its own, but there are\na lot of ways to create an infinite space and we may not detect all of them.\n\n\nAnother common scenario is websites which provide for filtering a set of search results in many\nways. A shopping site might allow for finding clothing items by filtering on category, price,\ncolor, brand, style, etc. The number of possible combinations of filters can grow exponentially.\nThis can produce thousands of URLs, all finding some subset of the items sold. This may be\nconvenient for your users, but is not so helpful for the Googlebot, which just wants to find\neverything\\&mdashl;once!\n\nCorrecting infinite space issues\n--------------------------------\n\n\nOur\n[Webmaster Tools Help article](https://www.google.com/support/webmasters/bin/answer.py?answer=76401)\ndescribes more ways infinite spaces can arise, and provides recommendations on how to avoid the\nproblem. One fix is to eliminate whole categories of dynamically generated links using your\nrobots.txt file.\n[The Help Center has lots of information on how to use robots.txt](/search/docs/crawling-indexing/robots/intro).\nIf you do that,\n[don't forget to verify that Googlebot can find all your content](https://www.google.com/webmasters/tools/robots-testing-tool)\nsome other way. Another option is to block those problematic links with a `nofollow`\nlink attribute. If you'd like\n[more information on `nofollow` links](/search/docs/advanced/guidelines/qualify-outbound-links),\ncheck out the Webmaster Help Center.\n\nWritten by Torrey Hoffman, Webmaster Tools team"]]
