Is My Website Banned in Google?

April 7th, 2011

One of the more difficult problems to diagnose is has your website been banned or is there something else wrong? I have the unfortunate task of trying to diagnose the problem for one of my sites and I thought it might be educational to let you in on the process.

First indicator a radical drop in traffic, especially from Google

traffic-drop

off to Google to check some rankings nothing’s listing. Next we do some standard checks:

site:example.com – sites mentioning the site but nothing from the domain in question
site:www.example.com – sites mentioning the site but nothing from the domain in question
inurl:example.com – nothing
inurl:www.example.com – nothing
link:example.com – returns results ***
link:www.example.com – returns results ***

Now the fact that the link command has results gives me some indication it’s not banned. This site was not part of Google’s sitemaps program so I submitted it, wondering if I would actually get some indication, the screen below is what I saw:

Am I Banned

Not a lot of help there. It would really be helpful if there was some little message that said something like “this site has been removed from the index for violating google quality guidelines” or at least if you knew with 100% certainty that it would be there if you were in fact banned. I get why you can’t tell me why I was banned, but at least tell me that I was banned.

Looking at the new crawl rate graph here’s what we get

Google Crawl rate

So it looks like Google can get in and is crawling. However looking at my site reports the only two pages crawled after October 6th are the robots.txt file which gets checked every few days, and the main page for the blog which gets checked every day or two, both were last checked on the 15th. Now if it was just the robots.txt file again I’d lean towards a ban, but the blog page is giving some conflicting data. I open up the robots.txt to check that it wasn’t hacked and changed and that looks ok and the htaccess file looks good too. I blow the dust off internet explorer with the google toolbar and I see a page rank three. Usually when you’re banned you’ll see 0 or graybar so more conflicting data. Lastly I don’t see any of the chilling effects “some sites have been ommitted” links at the bottom of the SERP’s.

I haven’t told you much about the site up until now, and that’s actually on purpose. The site’s been running since 2005, there’s maybe a little bit of dupe content and some slightly aggressive linking, but nothing too heinous. The site does run adsense and was actually a site selected to be part of the Google Adsense Video Beta program, and got me a cool lava lamp. So that does mean some people looked at the site but I don’t think they would have approved it for the program if it was spam central, and I don’t think there’s a connection between that and the site dropping (must resist tin foil hat).

So I don’t think the site was banned but it’s hard to say for certain, but the lack of anything under the URL and complete loss of rankings is odd. I’m going to make a sitemap and submit it and see what happens. If I don’t see things start to return to normal next week I’ll probably move hosts and see if that’s an issue.

UPDATE:
So I submitted the sitemap last night before going to bed and here the new sitemaps diagnostic page

sitemapsupdate II

and Here’s the summary page

sitemapsupdate1

Running my stats program I can see they visited the homepage and the main blog page, again not the behavior I would expect to see on a banned site

robotsupdate

Update III
So I’ve gotten some data out of Google Sitemaps, which is very interesting …
url-blocked

They tell me 33 URL’s are blocked in the robots.txt file. As I mentioned above I checked the robots.txt file and htaccess file both of which looked ok and had the original date and timestamps on them. I grabbed a handful of URL’s and submitted them through the robots analysis tool and the URL’s that were blocked are now OK. At this stage my two best working theories are as follows:

a) there was some technical issue at the host and the wrong files somehow got served creating an “issue” with the robots.txt file

b) Someone hacked into my account uploaded a bogus robots file and submitted the site to the URL removal tool. Now if they were a clever chap they would have saved the original robots file and put it back once the damage was done. Unfortunately this host doesn’t give me access to the FTP logs so I can’t verify that action. I have however changed the login as a precaution.

If that was the case it’s probably going to take quite a while to get all 400 pages back in the index, which is kind of a bummer as we are coming into to “prime season” for this website.

Update IV
The adsense bot has been all over the popular pages all day. I have seen the standard Googlebot on site three times, twice to the home page and once to a deep page, within the past 12 hours. It would be nice if the media bot fed the pages into the index, I don’t expect to see anything for 4-5 days will keep this thread updated.

UPDATE V
Well I’ve made some progress Goolge only thinks 16 pages are blocked via robots.txt down from 33. Sadly no pages are in the index yet

sm-3

Why has Google banned my website?

April 7th, 2011

As an SEO company this is one question we get very often. There are few bigger problems for an Internet business or search engine marketer than to find that their website has disappeared from Google’s search rankings. Sometimes their website doesn’t even rank for their own web site’s name. How did this happen? Read below to find out some of the reasons why a site may have been banned from Google and what to do to get back in the rankings.

Usually there is no warning for being banned or penalized by Google except for the steady drop of sales and visitors to your site. Many site owners and search engine optimization firms are left with little to no idea why they were removed and can be left scratching their heads as to how to get back in. While there are many reasons why a site has been banned, here are a few of the more common reasons. If your site has been banned contact your SEO company or give Big Oak a call to help you get back on the right track to high Google rankings.
1. Robots and Meta Tags

The first and simplest solution many be that your robot.txt file has been changed to prevent search engines from entering your site. Or your meta tags could be directing the search engine robots to exclude your site. While this would be highly unlikely, it is best to rule this out. So check your robot.txt file (if you have one) and your meta tags. Unless you want your site hidden, you should never read this in your meta tags: . If you see this, you are blocking your site from Google.

You can also ban your own site by having a robots.txt with the wrong code. Two examples of robots.txt code are below.

This example allows all robots to visit all files because the wildcard “*” specifies all robots.

User-agent: *
Disallow:

This example keeps all robots out:

User-agent: *
Disallow: /

Read more about this at: http://en.wikipedia.org/wiki/Robots.txt
2. Cloaking (A Big Google No-No)

Straight from Google’s website: “The term “cloaking” is used to describe a website that returns altered web pages to search engines crawling the site. In other words, the web server is programmed to return different content to Google than it returns to regular users, usually in an attempt to distort search engine rankings. This can mislead users about what they’ll find when they click on a search result. To preserve the accuracy and quality of our search results, Google may permanently ban from our index any sites or site authors that engage in cloaking to distort their search rankings.”

If your website or web pages are set up to display different information  for a search engine spider versus a real person, then you are cloaking. Cloaking delivers one version of a page to an Internet user and a different version to a search engine. The cloaked page is packed with keyword and terms that the site wants to be highly rank for so, in essence, they are cheating. There are good reasons for cloaking as well, such as targeted advertising, but if you are trying to manipulate your rankings you should put an end to this immediately.
3. Duplicate Content or Websites

If Google finds multiple web pages have the same content they may penalize each website for this. Of course, someone may have copied your content and Google banned you even though it was your original content that was taken. Make sure no other site is using your content. You can do this by performing a Google search using some of your text with quotation marks (”) around it. If you do find someone is using your original copy visit here to learn more about copyright infringement: http://www.google.com/dmca.html.

You can check here to see if your site has been duplicated unbeknowst to you: http://www.copyscape.com
4. Hidden Text and or Links

How can text been hidden? Well, there are a variety of ways – some are more sneaky than others. But is boils down to this: it is considered hidden if the text or link is invisible to the website visitor but can be seen by search engine spiders. This used to be done quite often, such as making your text white on a white background or using cascading style sheets (CSS) to hide your text, but search engines can easily spot this today so it is best to avoid it altogether.
5. Keyword Spam and Keyword Stuffing

Ever seen a web page with a very awkwardly written first paragraph where a certain word is repeated ad nauseam? Here’s an example:

“We sell the best father’s day gifts for father’s day. If you like to celebrate father’s day we can help with the best father’s day gifts for father’s day.”

Care to guess which keywords are being targeted? This is keyword spamming or stuffing but it is just the tip of the SEO iceberg. This is just the content on the page, there is probably keyword stuffing happening in the code: in the meta tags, invisible text, alt tags, title tags and comment tags. etc. If the word or phrase is repeated too often Google can place a filter to reduce the site’s rankings or simply ban the site. Keyword density can be tricky but, as a general rule, Big Oak shoots for 1% to 5% of all text on a page to be our targeted keywords. Ultimately you must write for the reader not the search engine. Be sure the keywords flow naturally.
6. Doorway Pages

Defining a doorway page can be difficult so here is our definition that could potentially ban your site in Google: pages that are created in order to attract search engine spiders and be ranked highly for their targeted keywords. Real visitors find this page and then continue to the “real” website from there. Hence the name “doorway page”. These pages aren’t in the navigation most of the time. If you come across a page where much of the information is duplicated from other pages on the site but it is different in terms of keywords only, this is most likely a doorway page.

As you can see this can be a gray area. Some pages on a website may focus on a particular subject and be innocent of trying to lure search engine spiders only for high rankings. Err on the side of caution and make sure the page is useful and part of the your site’s navigation.
7. Redirect Pages

Sneaky redirection pages are set up in groups from 5 to hundreds. They all target similar and related keywords or phrases. Usually, the only links on these pages are links to other pages in the same family creating a false sense of related linking.

These pages don’t necessarily contain content that any human would be interested in. These pages may show up high in Search Engine Results Pages (SERPS), but when you click on one of these pages from the SERPS, you will be redirected to another page. In other words, the page you click to see is not the page you actually get to read.

The redirect can be automatic, done with a meta refresh command or through other means such as a the mouse moving while on the redirect page.
8. Buying Links

While buying links may not get you banned, they can certainly hurt your page rank. Google has slowly been catching on to this fad and has measures in place to put your site in limbo for 6-8 months (known as the “sandbox effect”) so you can’t instantly benefit from buying links to your website. Many sites that sell links are being devalued by Google, making an investment in this strategy a waste of money and time. Ultimately, stay away from buying links to increase your ranking.
9. Linking to Bad Neighborhoods

Link campaigns are good thing when done correctly; we would say they are a necessity in today’s SEO world. But linking to bad neighborhoods are a sure way to lose your rank in Google. If you aren’t careful about who you are linking to you can easily disappear overnight. Basically, while you may be ethical and do everything right linking to someone who isn’t can be considered guilt by association. Always verify your links to other sites. Make sure they have page rank in Google and are indexed by Google. Try searching for their URL to see if they are indexed. Avoid linking to any sites that use spamming techniques to increase their search engine rankings. Regularly checking outbound links from your site and removing any offenders is a good idea.

A few site types to avoid:

* Free-for-all link farms
* Adult sites
* Gambling sites

10. Code swapping

Optimizing a page for top ranking, then swapping another page in its place once a top ranking is achieved.
What does Google say?

“Don’t deceive your users, or present different content to search engines than you display to users,” Google says, and they list some bullet points on avoiding being banned.

* Avoid hidden text or hidden links.
* Don’t employ cloaking or sneaky redirects.
* Don’t send automated queries to Google.
* Don’t load pages with irrelevant words.
* Don’t create multiple pages, subdomains, or domains with substantially duplicate content.
* Avoid “doorway” pages created just for search engines, or other “cookie cutter” approaches such as affiliate programs with little or no original content.

Google also states:

“Avoid tricks intended to improve search engine rankings. A good rule of thumb is whether you’d feel comfortable explaining what you’ve done to a website that competes with you. Another useful test is to ask, ‘Does this help my users? Would I do this if search engines didn’t exist?’”

While creating a page without a thought to search engines is probably going a little too far, optimizing your site for an organic search, as long as it conforms to their standards, is perfectly acceptable.
How to get back into Google

Visiting this old link I used to reccommend http://www.google.com/support/bin/request.py shows no option for “I’m a webmaster inquiring about my website” any longer which allowed you to request reinstation.

However, logging in to Google Sitemaps now shows a direct link at the bottom of your main account page to “Submit a reinclusuion request” which takes you here – https://www.google.com/webmasters/sitemaps/reinclusion?hl=en

This means you will have to register your site to do.

From there you get to check boxes that let you admit guilt, acknowledge modification, agree not to do it again, and even a box to explain yourself.

You don’t have to contact Google but it can’t hurt. They will eventually spider your site again and see that you have cleaned up your website. You may have to wait a few months for Google to re-index your site so be patient and don’t tinker with your website too much unless dictated by your site’s products or content needs.

The worst case scenario is to start a new site. Sometimes this can be necessary but only in the most extreme cases.

phpLD 3.2 and 3.0.6 Template Released

February 18th, 2008

I am here to announce the release of a phpLD 3.2 and 3.0.6 template.

DemoDirectoryCache

Download version 3.2 | version 3.0.6

Template was provided by Templates Factory & Directory Journal Web Directory

Thanks,

Gotcha Directory Releases a phpLB 1.4 template

January 10th, 2008

Since phpLinkBid released their much awaited 1.4 version back in late 2007, theme designers haven’t been very quick to get their templates upgraded or new ones released. The new features in v 1.4 are far superior to earlier versions so people are willing to use the default template in order to get the most out of the script and run 1.4

I am proud to announce that Gotcha Web Directory has released a new template made free by Autority Business Directory. The template uses 2 columns and unlike many others, has the categories and linkroll placed in the left sidebar.

The template is free to use providing footer links remain intact. Theme is table-less so is SEO friendly as it contains a better code : text ratio.

Preview / Download Theme

Keep checking our phpLB templates page for other templates as we have a few more to release. The aurora theme is available to download for phpLinkDirectory also in our phpLD templates area.

Tracking Directory Submissions with Google Analytics

December 31st, 2007

A paid submission to your directory is great. It means you get paid and your marketing efforts are working.

More going out than coming in?

What happens if you are spending thousands a month marketing your site, yet getting only a small percent of that back in paid submissions? Are some of your monthly costs going to waste? Does the traffic result in no submissions? Is it worth paying all that money for the low quality traffic you do get?

Find the source of these submissions

The best way to answer your questions is to know how your customers reach your directory. Knowing the source of the sale can help you control your monthly spending, know where to advertise other directories you may have, and know what areas you need to focus your attention to.

How can I track them?

Tracking your submissions is really easy. It doesn’t require any programming knowledge nor does it cost any money. It can be done using Google Analytics – a free traffic analysis program created by google. Go to the site and check it out. You will need a google account which you may already have if you use one of their other services like GMail. Google Analytics tracks your visitors. Where they came from, what pages they view, how long they stay on the site and hundreds of other useful information is collection.

I now have analytics running on my site

Its really easy to add the analytics code to your site. If using phpLD, edit the footer.tpl file and place the code given to you before the </body> tag. You have probably noticed how useful this software is. It shows where your traffic comes from, what pages they view, what keywords they used in search engines and much more. Onward we go!

I have seen that section called “goals”, but have no idea what its for

You are not alone. Many webmasters just ignore the goals feature. It is of no use to some. Goals however is what we will use to find the source of our paid submissions in our directory.

Enough of the rubbish, tell me how to do it

I really do talk too much. I’ll get right to the juicy bits now. How to create a goal and start tracking those useful submissions.This is currently set up for use with phpLinkDirectory.
Step 1 – Login to your Analytics account and stop at the page where your website profiles are listed.

(Click image for larger preview)

Step 2 – Next to the website profile of your directory, click edit. You will be taken to an overview of the website profile.

Step 3 – There is a section listing goals. One the first line (G1) click edit. See image below.

(Click image for larger preview)

Step 4 – You will be given a form to enter details of the goal. In our case, a goal is a paid submission.

Set the goal to active.

Give it a “head match”.

Set the “Goal URL” to “http://www.yoursite.com/payment.php?payed=” (without the the quote marks).

Give the goal a name. I called mine “submissions”.

Goal value should be the amount your regular submissions cost. I typed “20″. No currency symbol is needed.

All settings are shown in image below.

(Click image for larger preview)

Step 5 – Add funnels. Funnels show how the user navigated through the site to reach payment. Its fairly straight forward – they will all come throught the submit form.

For step 1 use URL “http://www.yoursite.com/submit.php” and name it “submit”.

Step 2 should be named “Payment” and the URL “http://www.yoursite.com/payment.php?id=” should be used.

Step 6 - Click save. Thats it!

Does it actually work?

I never thought it would actually work. Yesterday I announced my directory on digital point forums, so logged in this morning to see if my 2 submissions got tracked.

Choose todays date. The other days should be greyed out. Its not vital but it gives the ability to view one day at a time. See screenshot below for help on viewing one day only.

(Click image for larger preview)

Click the goals button

The above image shows I had 2 conversions which was correct. They were shown on the date I got the submissions (December 30th). See screenshot below for info.

(Click image for larger preview)

On the left navigation bar, click traffic sources. Still viewing December 30th. Choose one of your traffic sources or select all traffic sources. As seen in the screenshot below, the “Goal Conversion” tab must be clicked. It shows how your traffic sources resulted in conversions. In my case, 47 visits from Digital Point Forums resulted in 4.26% of them submitting a site. This was highlighted in the screenshot below.

(Click image for larger preview)

Conclusion

There we have it! A solution to your marketing puzzles. Now you can find where your customers came from you will know which method of your marketing is working best.

Feel free to contact me with any issues you have or any problems with installing analytics on your site and setting up goals.

WordPress Themes Centre

October 28th, 2007

Some good news!

We have completed the implementation of our free themes for wordpress resources area. The section is similar to the phpLD themes, but with some improvements. The downloads of each theme can be seen on the main page. More info on a theme can be seen on the download page.

We have also added a demo wordpress blog where all themes can be seen and tested so a user can see the template in action before downloading it.

You can check out the free wordpress themes here – free wordpress themes.

~Mike

Welcome to the blog

September 16th, 2007

Hi guys,

As usual, i make a first post to kick things off. This time round we will have plenty webmaster and phpLD resources for you all.

I expect to blog around twice a week, so mkae sure you get subscribed – it saves you the hardship of coming back here to check for updates!