Friday, March 7, 2025
A long-standing tool for website owners, robots.txt has been in active use for over 30 years and is broadly supported by crawler operators (such as tools for site owners, services, and search engines). In this edition of the robots refresher series, we'll take a closer look at robots.txt as a flexible way to tell robots what you want them to do (or not do) on your website.
Getting started with robots.txt
The way these files work is simple: you make a text file called "robots.txt" and then upload it to your website—and if you're using a content management system (CMS), it's likely even easier. You can leave your robots.txt file empty (or not have one at all) if your whole site may be crawled, or you can add rules to manage crawling. For example, to tell all bots (also known as crawlers, robots, spiders) to stay out of your "add to cart" page, you could write this in your robots.txt file:
user-agent: * disallow: /cart
More specific things you can do with robots.txt
robots.txt is the Swiss Army knife of expressing what you want different robots to do or not do on your website: it can be just a few lines, or it can be complex with more elaborate rules targeting very specific URL patterns. You can use a robots.txt file for solving technical issues (such as unnecessary paginated pages), or for editorial or personal reasons (such as just not wanting certain things crawled). For example, you could:
Inform multiple bots (but not all) about the same rule
This group tells both |
user-agent: examplebot user-agent: otherbot disallow: /search |
Tell one bot to avoid paths that contain a specific piece of text
For example, you could tell |
user-agent: documentsbot disallow: *.pdf |
Tell a bot it may crawl your blog, but not the drafts |
user-agent: documentsbot allow: /blog/ disallow: /blog/drafts/ |
Block a crawler from part of your website, while allowing other crawlers to access your site
This robots.txt file disallows the mentioned |
user-agent: * allow: / user-agent: aicorp-trainer-bot disallow: / allow: /$ |
Leave a comment for your future self
You can start a line with |
# I don't want bots in my highschool photos user-agent: * disallow: /photos/highschool/ |
For even more, you can check out our list of useful robots.txt rules.
Making changes to your robots.txt file (practically)
The Robots Exclusion Protocol (REP) works by putting together rules ("allow" or "disallow") and specifying which robots these rules apply to. You don't need to learn programming or fiddle with tools; you can just put these rules in a text file and upload it to your website.
For most websites, it's even simpler than that! If you're using a CMS, it usually has something already built in to help you change your robots.txt file. For example, some CMSes let you customize your robots.txt file using checkboxes or with a simple form, and many have plugins that help you set up and write rules for your robots.txt file. To check what's possible within your CMS, you can do a search for the name of your CMS +"edit robots.txt file".
Once you've got things set up, you can also test to make sure your file is set up how you intended. There are many testing tools built by the web community to help with this, such as TametheBot's robots.txt testing tool and this robots.txt parser that are using the open-source robots.txt parser library.
If you have any questions about robots.txt, you can find us on LinkedIn or chat with like-minded experts in our community forums.