RSS Feed3 ways to clean up your Google search results
February 9th, 2008 by Verne
It’s not quite Spring yet, but we’ve been spending time doing some cleaning. If you’ve owned a server for several years, you’ll know that over time things can get messy. In fact, if you do a quick Google site search (site:www.yourdomain.com), what you end up seeing is a long, and sometimes ugly, history of every page and file you’ve ever publically offered.
A big part of our recent new site launch was cleaning up our old links to ensure that our audience was finding the most relevant content in major search engines like Google. Our server currently hosts thousands of pages and files - some we publish for the public, some we like to keep private. Targeted campaign pages, internal or upcoming project sites, and client work are examples of areas of our server that we’d rather not have someone find while doing a search query.
Knowing this, we’ve implemented a few tools to help keep our search results clean and tidy. Below you’ll find 3 simple steps to follow that will help you do the same.
1. META Tags
Here’s the first place you want to start. You can add a simple META tag to the top of every live page you have (between the <head> and </head> tags) and configure it to your liking. Here’s how the META tag should look:
<meta name="robots" content="noindex, nofollow">
noindex signifies that site crawlers should not index the page. Alternatively, writing index would tell crawlers to index the page.
nofollow signifies that links on this page should not be tracked or given credit for (alternatively, follow would track links). nofollow is also used commonly for individual links with the rel="nofollow" attribute when web publishers don’t want to get punished for linking to suspicious sites.
By default, every page is tagged as “index, follow”, but changing this attribute can help you configure your pages in a few different ways (4 to be exact).
WordPress Meta Robots Plugin
For WordPress users, you can install the Meta Robots plugin. This will allow you to configure each and every post and page from the editor, as well as configure the global settings of your site.
2. Robots.txt
Tagging each and every one of your pages with META tags can be tedious. Sometimes, it makes sense to simply block an entire directory from being crawled. To do this, create a file, name it robots.txt, and upload it into the root of your server (i.e. it should reside at www.yourdomain.com/robots.txt).
The contents of your robots.txt file should look something like this:
User-agent: *
Disallow: /private/
Disallow: /internal/
User-agent refers to the specific site crawler bots that you would like to block from accessing your directories. For example, you may want to only block Google’s bots but allow all others. In this case, you would write User-agent: Google (the name of each bot varies and will require some additional research to find it). A * means that you are targeting all bots.
Beneath user-agent, you can list out each directory you would like to have blocked. When a directory is blocked, so are all of its subdirectories, so be careful!
Note that a robots.txt file can also block individual pages and files, so it can be used as a substitute to using META tags.
3. Google Webmaster Tools
Thus far I’ve only shown you ways to avoid having new pages or directories picked up by search engines. If your pages have already been indexed then you’ll need to take one extra step to have it removed. That’s where Google Webmaster Tools comes into play.
Google Webmaster Tools is a great place for site owners to manage how their site is being indexed by Google. There are a multitude of different features to this site, but I’ll only be covering one particular one - the Remove URLs feature. However, if you are a site owner, I strongly encourage you to sign up to this free service and browse around.
Once you’ve registered/signed-in and verified that you are the actual site owner, click Tools and then Remove URLs on the left navigation.
You’ll notice at the top of the page, there are one of three things you must do before being able to submit a Removal Request. For a directory/page/file to be removed from Google’s index, you must do one of the following:
- Make sure the content is no longer live on the web. Requests for the page or image you want to remove must return an HTTP 404 (not found) or 410 status code.
- Block the content using a meta noindex tag.
- Block the content using a robots.txt file.
In other words, you must either delete the directory/page/file from your server, or do one of the 2 things I’ve already discussed.
Once this has been done, go ahead and click the New Removal Request button. You’ll find that Google gives you 4 options to choose from:
Choose an option and submit your request. Your request will be added to a Pending list of requests, and approximately 24 hours later, if all goes well and all requirements have been satisfied, you’ll find that your requests will have been fulfilled and your directory/page/file will have been removed from Google’s index.
Closing Notes
This tutorial is focused on only cleaning up your search results on Google. While it is the leading search engine today, there are a number of other major engines to also share your attention with.
In any case, cleaning up your search results is an important maintenance task for all site owners as it helps you ensure that your audience will only find content that’s current and relevant. Performing the 3 simple tasks in this tutorial on a regular basis will greatly help you take control over what people are finding on your site and help clean up your search results.
