Make Sure You Aren’t Blocking Google

Just a quick blog post, unfortunately with another cautionary tale.

Always make sure that your site isn’t stopping Google from indexing it’s pages. This tends to happen when a site has been redeveloped and the restricing code is left in when the development site is moved over to the live site. It’s not uncommon, but usually spotted very quickly, or atleast in most cases it should be.

Last week I was with a company that were unaware that their site had a robots.txt file with the code disallowing Google from indexing ALL the content on their pages. Oops!

Quick way to check that your site is ok;

i) Do a site operator check in Google. Just go to Google, type in site:yourwebsitename.com (leave out the www, and replace yourwebsitename with your competitors domain)

If you only see one reference, rather than a list of pages, then you’re probably in trouble. Do a check with your competitors and see what their sites return.

Google may also show the message ‘A description for this result is not available because of this site’s robots.txt’ and display 1 result.

Here’s an example of a site using their robots.txt file to block Google. You’ll see very few pages indexed for this domain.

web archive robots file blocking search engines

ii) Check the Organic results in Google Analytics. If you’re getting no traffic from non-paid search, then this could be the issue.

iii) Check ‘Index Status’ in Google Webmaster Tools. This will give you an idea of how many pages are indexed in Google.

I hope to have another article shortly on 10 or more of the biggest SEO mistakes that I commonly see.



Comments are closed.

Michael Wall

Michael Wall is an experienced web developer based in Belfast N.Ireland currently running his own web design company.

Michael Wall is available for Internet Marketing, Google Adwords PPC and SEO work. Please call Belfast 02890 923383 or use the contact form.

Google Partner

Popular Posts

SEO training in Belfast