URL removals explained, part II: Removing sensitive text from a page

Friday, August 06, 2010

Change can happen—sometimes, as we saw in our previous post on URL removals, you may completely block or remove a page from your site. Other times you might only change parts of a page, or remove certain pieces of text. Depending on how frequently a page is being crawled, it can take some time before these changes get reflected in our search results. In this blog post we'll look at the steps you can take if we're still showing old, removed content in our search results, either in the form of a "snippet" or on the cached page that's linked to from the search result. Doing this makes sense when the old content contains sensitive information that needs to be removed quickly—it's not necessary to do this when you just update a website normally.

As an example, let's look at the following fictitious search result:

To change the content shown in the snippet (or on the linked cached page), you'll first need to change the content on the actual (live) page. Unless a page's publicly visible content is changed, Google's automatic processes will continue to show parts of the original content in our search results.

Once the page's content has been changed, there are several options available to make those changes visible in our search results:

1. Wait for Googlebot to re-crawl and re-index the page: This is the natural method for how most content is updated at Google. Sometimes it can take a fairly long time, depending on how frequently Googlebot currently crawls the page in question. Once we've re-crawled and re-indexed the page, the old content will usually not be visible as it'll be replaced by the current content. Provided Googlebot is not blocked from crawling the page in question (either by robots.txt or by not being able to access the server properly), you don't have to do anything special for this to take place. It's generally not possible to speed up crawling and indexing, as these processes are fully automated and depend on many external factors.

2. Use Google's public URL removal tool to request removal of content that has been removed from someone else's webpage. Using this tool, it's necessary to enter the exact URL of the page that has been modified, select the "Content has been removed from the page" option, and then specify one or more words that have been completely removed from that page.

   

   Note that none of the words you enter can appear on the page; even if a word has been removed from one part of the page, your request will be denied if that word still appears on another part of the page. Be sure to choose a word (or words) that no longer appear anywhere on the page. If, in the above example, you removed "top secret velocitus incalculii capturing device", you should submit those words and not something like "my project." However, if the word "top" or "device" still exists anywhere on the page, the request would be denied. To maximize your chances of success, it's often easiest to just enter one word that you're sure no longer appears anywhere on the page.

   Once your request has been processed and it's found that the submitted word(s) no longer appear on the page, the search result will no longer show a snippet, nor will the cached page be available. The title and the URL of the page will still be visible, and the entry may still appear in search results for searches related to the content that has been removed (such as searches for velocitus incalculii), even if those words no longer appear in the snippet. However, once the page has been re-crawled and re-indexed, the new snippet and cached page can be visible in our search results.

   Keep in mind that we will need to verify removal of the word(s) by viewing the page. If the page no longer exists and the server is returning a proper 404 or 410 HTTP result code, making us unable to view the page, you may be better off requesting removal of the page altogether.

3. Use Google Webmaster Tools URL removal tool to request removal of information on a page from your website. If you have access to the website in question and have verified ownership of it in Google Webmaster Tools, you can use the URL removal tool there (under Site Configuration > Crawler access) to request that the snippet and the cached page be removed until the page has been re-crawled. To use this tool, you only need to submit the exact URL of the page (you won't need to specify any removed words). Once your request has been processed, we'll remove the snippet and the cached page from search results. The title and the URL of the page will still be visible, and the page may also continue to rank in search results for queries related to content that has been removed. After the page has been re-crawled and re-indexed, the search result with an updated snippet and cached page (based on the new content) can be visible.

Google indexes and ranks items based not only on the content of a page, but also on other external factors, such as the inbound links to the URL. Because of this, it's possible for a URL to continue to appear in search results for content that no longer exists on the page, even after the page has been re-crawled and re-indexed. While the URL removal tool can remove the snippet and the cached page from a search result, it will not change or remove the title of the search result, change the URL that is shown, or prevent the page from being shown for searches based on any current or previous content. If this is important to you, you should make sure that the URL fulfills the requirements for a complete removal from our search results.

Removing non-HTML content

If the changed content is not in (X)HTML (for example if an image, a Flash file or a PDF file has been changed), you won't be able to use the cache removal tool. So if it's important that the old content no longer be visible in search results, the fastest solution would be to change the URL of the file so that the old URL returns a 404 HTTP result code and use the URL removal tool to remove the old URL. Otherwise, if you chose to allow Google to naturally refresh your information, know that previews of non-HTML content (such as Quick View links for PDF files) can take longer to update after recrawling than normal HTML pages would.

Proactively preventing the appearance of snippets or cached versions

As a webmaster, you have the option to use robots meta tags to proactively prevent the appearance of snippets or cached versions without using our removal tools. While we don't recommend this as a default approach (the snippet can help users recognize a relevant search result faster, and a cached page gives them the ability to view your content even in the unexpected event of your server not being available), you can use the "nosnippet" robots meta tag to prevent showing of a snippet, or the "noarchive" robots meta tag to disable caching of a page. Note that if this is changed on existing and known pages, Googlebot will need to re-crawl and re-index those pages before this change becomes visible in search results.

We hope this blog post helps to make some of the processes behind the URL removal tool for updated pages a bit clearer. In our next blog post we'll look at ways to request removal of content that you don't own; stay tuned!

As always, we welcome your feedback and questions in our Webmaster Help Forum.

Other posts of this series

• Part I: Removing URLs and directories
• Part II: Removing and updating cached content
• Part III: Removing content you don't own
• Part IV: Tracking requests, what not to remove

Finally, you might be also interested to read about managing what information is available about you online.

Posted by John Mueller, Webmaster Trends Analyst, Google Switzerland