Tuesday, March 06, 2007
Dan Crow has written about this on the Google Blog recently, including an introduction to setting up your own rules for robots and a description of some of the more advanced options. His first two posts in the series are:
While we're on the topic, I'd also like to point you to the robots section of our help center and our earlier posts on this topic:
Update: For more information, please see our robots.txt documentation.