I love the start of a new month. Its like you get to start over in some ways. For me my stat programs run monthly for the most part so its cool to see new data.
For instance – Not to long ago I made a post about how I knew I was in supplemental hell. Well Aaron Wall made a list of stuff for me to do to fix it (mostly in my robots.txt) and I was skeptical but I am happy to report not only am I out of supplemental hell but my Google traffic has increased 1400% in only 1 month after implementing his list of stuff.
Anyway Welcome February!
Ricky Peterson (24) 
ZK @ Web Marketing Blog (23) 
Holly Mann (12) 
R Kumar (12) 
gurtey (9) 
Shanker Bakshi (9) 
Jacques Snyman | 3 Quotes (8) 
Yisraeli Foods (8) 
Find Affiliate Offers (5) 
Michael Zhao (77) 
almir (76) 
fas (75) 
R Kumar (74) 
Rick Kats (70) 
Jay @ work from home (62) 
Dino (61) 
Dick (628) 
Goran Website (495) 
Melvin (330) 
Ben Pei (329) 
Taris Janitens (274) 
meethere (269) 
Tushar Dhoot (265) 
Tushar (260) 
Moneybites (259) 
Thanks for the comments….
Just curious if you know of any other case studies with implementing a robots.txt file for Google & getting large traffic increases on your wordpress blogs?
I’m playing with a couple of the many wordpress blogs I have. These particular 2 are both 99% supplemental according to this tool:
http://www.mapelli.info/tools/supplemental-index-ratio-calculator/
It’s been about a month and stuff is SLOWLY coming out of supplemental. One is at 75%, the other 95% now.
Still no changes in traffic or rankings…
Think I’m going to be making some changes to my robots.txt file tonite…
especially this one, I have my birthday during february!
Does robots.txt override sitemap.xml that I submitted to Google? For instance, if in my sitemap.xml I have /page/2/ but in robots.txt I use disallow /page/, will it get skipped?
You can actually just use this in your .htacesss file:
Options -Indexes
That’ll stop anyone include spiders from crawling or access your file structure. Pretty good for securing folders too.
Stuart
Thanks for posting about the robots.txt tip, it got me thinking and realizing just how much useless garbage I have indexed on Google now that’s probably killing me. I’ve blocked some stuff across a few of my sites, hopefully that will bring the same results you saw.
Basically, you allow or disallow robots to index certain pages. Allow pages that are important (your posts), and disallow the ones that are not (ie archives pages,advertising page,search page, etc). By having good, original, unduplicated content indexed, you’ll have a better ranking. Check wikipedia for instructions on how to format robot.txt or using the robot meta tag: http://en.wikipedia.org/wiki/Robots.txt
SM, share the secrets!
Would the Head Meta Description do the trick for those dynamic description tags?
How to avoid supplemental hell, Shoe..? need tips plz..
That is a smart move. I wish I knew which pages I could take out of the sites I am taking care of to get traffic that would help me out.
Aaron explaions this on his blog today here:
http://www.seobook.com/archives/002021.shtml
If folks are still interested. In short there’s only so much link authority to go around and shoe had that authority spread too thin so he disallowed low $$$ pages to increase authority associated with high $$$ pages.
IMHO the 1400% makes this seem like a silver bullet… it is, like Aaron says a 20% improvement. These gains likely came from getting a couple authoritative, topical links which pulled him out of the supplemental. I don’t believe that these changes to the index file are in any way related to being pulled out of the supplemental index.
Hope this helps and congrats Jeremy
Hmm he didn’t say this site specifically is the one he’s talking about did he? Matt cutts said specifically that pagerank is what decides what is supplemental, so it makes sense that removing worthless pages would improve the ones that are left.
Thanks
anyone who gets a phd in cs – that’s a nerd.
course now that they have money, like those enron ceo crooked nerds, they can all go on those dirt biking, i’m a tough man safari’s in africa to prove they are NOT nerds. lol
speaking of nerds, guess i better start running my comments past my lawyers first.
John check out how pagerank distribution works. I used to have almost 40,000 pages indexed by google but nothing ranked in the top 10 for anything. Now that I have disallowed tons of content that was douplicate i have good rankings again… for instance look at serps script
heh…. have you ever met any Google “se nerds” if so you would know how rediculas that statement is.
Not that I know of.. you just edit the templates.
“Yeah I added dynamic titles and descriptions last month”
Is this an addon you can get for wordpress?
Shoe, are you going to have a “febrazy” month like me? Probably not
Aaron probably doesn’t have the same issues as his site is older and more established – Google is more lenient if your site is established.
Yeah I added dynamic titles and descriptions last month as well and its gotten a lot of blog out of supplemental. I’ll look to add the robots.txt and see how that turns out next month.
thanks for the tip Shoe
You wonder ? Give me a break, of course you can.
If you want to see if Shoes a big fat liar then why don’t you look at his results in the serps. See if they match what he has in the robots.txt. Make sure to check cache dates to.
…Investigate before you speculate…
thats about 50% the rest was adding dynamic descriptions/titles and doing a little onpage internal linking.
That list would be amazing to read!!!
*L* well i think one of the big keys that aaron wall suggested at Elite Retreat is not letting bots crawl thru useless pages like link pages and galleries, as with blogs or other sites we have pages that are more fun and nto really a gain in rankings.
wonder, can you cloak a robots.txt? show one to users, and one to the bots?
come on shoe, can be that easy:
User-Agent: Googlebot
Disallow: /link.php
Disallow: /gallery
Disallow: /gallery2
Disallow: /gallery2/
Disallow: /gallery/
Disallow: /category/
Disallow: /page/
Disallow: /pages/
Disallow: /feed/
Disallow: /feed
Maybe a combination of robots, and age of domain – just time to get out? or maybe you slipped a few shoemoney shirts to google’s top se nerds?
Anyone willing to explain this a little more?
LOL, I bet your robots.txt is going to be your most popular page today Shoemoney.
Love to see the list as well but part of the list can already be found when you add robots.txt behind the url.
Well, duh. Why didn’t we think of that? Although, the increase in traffic is probably just a result of getting out of supplementals.
Why does SEOBook robots.txt not have this?
Removing duplicate content and pages with little to no written content bring a lot of pages out of supplemental. The mroe pages your site has in supplemental the less ranking power your domain has. If you decrease your supplemental pages your domain’s pagerank and ranking power increase.
I dont get it.. its one thing to kill off duplicates, another entirely to get an extra 1400% in traffic ???
The bots file *reduces* the amount of content you have in the index. I dunno.. sounds like somethings missing here..
Good to hear you got it fixed. Oh and thanks for including me in your list Shoe…
if you guys want to know just look at the robots.txt contents:
User-Agent: Googlebot
Disallow: /link.php
Disallow: /gallery
Disallow: /gallery2
Disallow: /gallery2/
Disallow: /gallery/
Disallow: /category/
Disallow: /page/
Disallow: /pages/
Disallow: /feed/
Disallow: /feed
rly well done jeremy.
I love new month, too. It’s every time a new chance to earn more than before.
1400% increase and no supplementals sounds like another good reason to visit the elite retreat! Maybe you can offer a budget alternative with slightly less info – “Search Bootcamp” $13.99 E-book!
Excellent information.. thx Jeremy!
I have a few supplemental probs…would love to see the list
Yeah, share that list. We want tell.
Yeah, share the list with us. We want tell. lol
ahem…any chance of sharing that list