I love the start of a new month. Its like you get to start over in some ways. For me my stat programs run monthly for the most part so its cool to see new data.
For instance – Not to long ago I made a post about how I knew I was in supplemental hell. Well Aaron Wall made a list of stuff for me to do to fix it (mostly in my robots.txt) and I was skeptical but I am happy to report not only am I out of supplemental hell but my Google traffic has increased 1400% in only 1 month after implementing his list of stuff.
Anyway Welcome February!
- Comment Likes 
- Comment Dislikes
August 21, 2009 at 2:54 am
Great job.
It’s always good to make improvements and see the hard work pay off.
March 4, 2009 at 10:48 am
Thanks for the comments….
January 13, 2008 at 9:16 pm
Just curious if you know of any other case studies with implementing a robots.txt file for Google & getting large traffic increases on your wordpress blogs?
I’m playing with a couple of the many wordpress blogs I have. These particular 2 are both 99% supplemental according to this tool:
http://www.mapelli.info/tools/supplemental-index-ratio-calculator/
It’s been about a month and stuff is SLOWLY coming out of supplemental. One is at 75%, the other 95% now.
Still no changes in traffic or rankings…
June 26, 2007 at 4:13 pm
Think I’m going to be making some changes to my robots.txt file tonite…
April 9, 2007 at 3:17 pm
especially this one, I have my birthday during february!
February 19, 2007 at 12:45 pm
Does robots.txt override sitemap.xml that I submitted to Google? For instance, if in my sitemap.xml I have /page/2/ but in robots.txt I use disallow /page/, will it get skipped?
February 15, 2007 at 11:44 pm
You can actually just use this in your .htacesss file:
Options -Indexes
That’ll stop anyone include spiders from crawling or access your file structure. Pretty good for securing folders too.
Stuart
February 8, 2007 at 4:59 pm
Thanks for posting about the robots.txt tip, it got me thinking and realizing just how much useless garbage I have indexed on Google now that’s probably killing me. I’ve blocked some stuff across a few of my sites, hopefully that will bring the same results you saw.
February 7, 2007 at 3:48 pm
Basically, you allow or disallow robots to index certain pages. Allow pages that are important (your posts), and disallow the ones that are not (ie archives pages,advertising page,search page, etc). By having good, original, unduplicated content indexed, you’ll have a better ranking. Check wikipedia for instructions on how to format robot.txt or using the robot meta tag: http://en.wikipedia.org/wiki/Robots.txt
February 6, 2007 at 10:27 am
SM, share the secrets!
February 2, 2007 at 9:19 pm
Would the Head Meta Description do the trick for those dynamic description tags?
February 2, 2007 at 11:40 am
How to avoid supplemental hell, Shoe..? need tips plz..
February 2, 2007 at 9:27 am
That is a smart move. I wish I knew which pages I could take out of the sites I am taking care of to get traffic that would help me out.
February 2, 2007 at 7:58 am
Aaron explaions this on his blog today here:
http://www.seobook.com/archives/002021.shtml
If folks are still interested. In short there’s only so much link authority to go around and shoe had that authority spread too thin so he disallowed low $$$ pages to increase authority associated with high $$$ pages.
IMHO the 1400% makes this seem like a silver bullet… it is, like Aaron says a 20% improvement. These gains likely came from getting a couple authoritative, topical links which pulled him out of the supplemental. I don’t believe that these changes to the index file are in any way related to being pulled out of the supplemental index.
Hope this helps and congrats Jeremy
February 2, 2007 at 2:29 am
Hmm he didn’t say this site specifically is the one he’s talking about did he? Matt cutts said specifically that pagerank is what decides what is supplemental, so it makes sense that removing worthless pages would improve the ones that are left.
February 1, 2007 at 9:10 pm
Thanks
February 1, 2007 at 4:31 pm
anyone who gets a phd in cs – that’s a nerd.
course now that they have money, like those enron ceo crooked nerds, they can all go on those dirt biking, i’m a tough man safari’s in africa to prove they are NOT nerds. lol
February 1, 2007 at 4:28 pm
speaking of nerds, guess i better start running my comments past my lawyers first.
February 1, 2007 at 4:15 pm
John check out how pagerank distribution works. I used to have almost 40,000 pages indexed by google but nothing ranked in the top 10 for anything. Now that I have disallowed tons of content that was douplicate i have good rankings again… for instance look at serps script
February 1, 2007 at 3:20 pm
heh…. have you ever met any Google “se nerds” if so you would know how rediculas that statement is.
February 1, 2007 at 3:17 pm
Not that I know of.. you just edit the templates.
February 1, 2007 at 3:13 pm
“Yeah I added dynamic titles and descriptions last month”
Is this an addon you can get for wordpress?
February 1, 2007 at 2:23 pm
Shoe, are you going to have a “febrazy” month like me? Probably not
February 1, 2007 at 2:14 pm
Aaron probably doesn’t have the same issues as his site is older and more established – Google is more lenient if your site is established.
February 1, 2007 at 1:59 pm
Yeah I added dynamic titles and descriptions last month as well and its gotten a lot of blog out of supplemental. I’ll look to add the robots.txt and see how that turns out next month.
thanks for the tip Shoe
February 1, 2007 at 1:57 pm
You wonder ? Give me a break, of course you can.
If you want to see if Shoes a big fat liar then why don’t you look at his results in the serps. See if they match what he has in the robots.txt. Make sure to check cache dates to.
…Investigate before you speculate…
February 1, 2007 at 1:53 pm
thats about 50% the rest was adding dynamic descriptions/titles and doing a little onpage internal linking.
February 1, 2007 at 12:07 pm
That list would be amazing to read!!!
February 1, 2007 at 11:44 am
*L* well i think one of the big keys that aaron wall suggested at Elite Retreat is not letting bots crawl thru useless pages like link pages and galleries, as with blogs or other sites we have pages that are more fun and nto really a gain in rankings.
February 1, 2007 at 11:28 am
wonder, can you cloak a robots.txt? show one to users, and one to the bots?
come on shoe, can be that easy:
User-Agent: Googlebot
Disallow: /link.php
Disallow: /gallery
Disallow: /gallery2
Disallow: /gallery2/
Disallow: /gallery/
Disallow: /category/
Disallow: /page/
Disallow: /pages/
Disallow: /feed/
Disallow: /feed
Maybe a combination of robots, and age of domain – just time to get out? or maybe you slipped a few shoemoney shirts to google’s top se nerds?
February 1, 2007 at 11:19 am
Anyone willing to explain this a little more?
February 1, 2007 at 10:49 am
LOL, I bet your robots.txt is going to be your most popular page today Shoemoney.
February 1, 2007 at 8:56 am
Love to see the list as well but part of the list can already be found when you add robots.txt behind the url.
February 1, 2007 at 8:21 am
Well, duh. Why didn’t we think of that? Although, the increase in traffic is probably just a result of getting out of supplementals.
February 1, 2007 at 8:17 am
Why does SEOBook robots.txt not have this?
February 1, 2007 at 8:11 am
Removing duplicate content and pages with little to no written content bring a lot of pages out of supplemental. The mroe pages your site has in supplemental the less ranking power your domain has. If you decrease your supplemental pages your domain’s pagerank and ranking power increase.
February 1, 2007 at 7:24 am
I dont get it.. its one thing to kill off duplicates, another entirely to get an extra 1400% in traffic ???
The bots file *reduces* the amount of content you have in the index. I dunno.. sounds like somethings missing here..
February 1, 2007 at 6:36 am
Good to hear you got it fixed. Oh and thanks for including me in your list Shoe…
February 1, 2007 at 4:11 am
if you guys want to know just look at the robots.txt contents:
User-Agent: Googlebot
Disallow: /link.php
Disallow: /gallery
Disallow: /gallery2
Disallow: /gallery2/
Disallow: /gallery/
Disallow: /category/
Disallow: /page/
Disallow: /pages/
Disallow: /feed/
Disallow: /feed
February 1, 2007 at 4:02 am
rly well done jeremy.
I love new month, too. It’s every time a new chance to earn more than before.
February 1, 2007 at 3:24 am
1400% increase and no supplementals sounds like another good reason to visit the elite retreat! Maybe you can offer a budget alternative with slightly less info – “Search Bootcamp” $13.99 E-book!
Excellent information.. thx Jeremy!
February 1, 2007 at 2:21 am
I have a few supplemental probs…would love to see the list
February 1, 2007 at 2:08 am
Yeah, share that list. We want tell.
February 1, 2007 at 2:07 am
Yeah, share the list with us. We want tell. lol
February 1, 2007 at 1:57 am
ahem…any chance of sharing that list