I never mess around with this stuff, but does duplicate content reduce how well your site ranks overall?

Some really good info, if you are not an SEO you are pretty darn close

I’m about to implement this:(how does it look?)
User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */comments
Disallow: /*?*
Disallow: /*?
Allow: /wp-content/uploads

Thanks for the tips shoe. A lot of people don’t realize how much duplicate content on your site can really hurt you.

Big Help! Thanks. This should help a bunch.

Very Nice post! We all know how many site’s leave this simple step out (like the ebook sales people, who u do a simple site: and u find the members download area). You put it out plain and simple!!! Don’t you find it funny how people who are non seo people like you, make more money then the seo people. LOL. Have a fantastic week Shoe and everyone else! Make that $$$$$

thx for the great tips for the robots.txt and wordpress blogs

Thanks for the excellent tips Shoe. One of my blogs had been performing amazingly until Google decided to hate it last week. These tips are just what I need to try and work out if it’s a duplicate content issue..

I disagree with this assessment on some level. Sure, you don’t want duplicate content and it will negatively impact your site, but using the robots.txt file to fix the problem wouldn’t be my way to go.

The robots file tells Google not to even crawl the page. A better scenario would be to use the meta noindex and follow. This tells Google not to index the page, but it can and will still accumulate link juice to pass it on (unless this page is a dead end, then it’s pointless).

See this interview with Matt from a few months ago for a little more in-depth conversation.

I have the all-in-one seo pack which applies noindex, nofollow meta tags on the actual archive/category/tag pages. I wonder if this is still worth doing but I guess it can’t hurt.

Thanks a lot for the tips Jeremy

Thanks for that, really need to get to grips with this robots stuff, I am sure it helps with SEO although don’t quite understand how.

thanks, very helpful

lol at all the spammy comments. “I totally agree with everyone” lol

This is very helpful! Thanks for sharing.

Disallow /category/ is a good one to add. Just make *extra* sure your Permalink structure isn’t set up to include “category” == otherwise nothing will be indexed.

To help reduce DC, I also recommend blocking the archives (just add a new line for each year your blog has been online)

# Block Duplicate Content from Archives
Disallow: /2006/
Disallow: /2007/
Disallow: /2008/

I also have this
Disallow: /*?*

instead of this;
Disallow: /*?

Thanks Shoe! I appreciate all you have done.

Blocking /category/ is a good one. Just need to be careful that your Permalink structure isn’t setup to include “category” — otherwise nothing will get indexed.

I also use the following to block the archives. Just add a new line for each year your blog has been online.

# Block Duplicate Content From Archives
Disallow: /2006/
Disallow: /2007/
Disallow: /2008/

One more is that I use;
Disallow: /*?*

instead of;
Disallow: /*?

Great tutorial - more of this please! No matter what you say, its pretty good SEO stuff.

Thank you for the tips.

Excellent post Jeremy! Every single day I’m learning something new from you it seems. Just the other day with the link rel= and now today with the robot.txt file.

I’ve just read the basics about the robot.txt file and never really thought much more into it. It’s good we have people like you helping us out along the way.

Thanks!
~Terry

Thanks for the tip. I guess I have the same question as someone above. How does duplicate content hurt your ranking? Is it a consequence of PR being spread across multiple pages - or is it just a case of being penalized for duplication? I’ll have to do more research. Thanks again.

yes ok, Thanks, use in 28 blogs maide in brasil

Interesting that the question mark doesn’t have to be escaped. Normally a question mark would be a RegEx meta character, but I just looked it up in the Google guidelines: a question mark is treaded as a regular character.
An important note: Not every crawler understands RegEx in the robots.txt. So you are “protecting” your sites against the major search engines, but not from normal bots. This is ok to avoid duplicate content, I guess.

OMG. Will this keep spammers from doing that obnoxious thing where they copy a whole journal entry (or the majority of one) into their fake blogs, making it look like they are quoting it (”Someone said something great over at blahblahblah dot com, ‘entire post here,’”) with no other content? Just to get on google and steal my links? I’m sure they’re using robots at some stage….

Great list of tips shoe … i can bet this helps alot.

[...] Wordpress robots.txt tips against douplicate content - ShoeMoney® Some useful tips on updating your robots.txt file to avoid duplicate content problems with Wordpress. (tags: seo wordpress) [...]

For smaller blogs this might not be the best thing to do when it comes to SEO. If implementing everything this way, you are relying on Google to find older posts (if they don’t have links to them) by going directly through the homepage. Requiring Google to go back 20 pages to find an article is a good way to end up in the supplemental index (which, of course they claim doesn’t exist anymore, but IMO it does).

Thanks for the list and explaining it. I need to add a robots.txt file to my blog.

Thanks for these tips - I hadn’t even thought of leveraging the robots file against duplicate content (much easier than disabling those features!). Thanks!

I shall have to take another look at my robots.txt!

Typo in the title? Or am i seeing things

Thank you for posting these tips for WordPress on the robot.txt file.

Thats great. But don’t you think you are getting off topic.

[...] Read more of this article at ShoeMoney.com [...]

Hello to all I just want to share my post regarding Robots.txt that really helps my site

Thanks! Very useful again. Avoiding duplicate content really helped me ranking well.

[...] Wordpress robots.txt tips against duplicate content [...]

Well, this seems to be a better version than all noindex plugins going arround. Def. will give it a try!

[...] citeva zile am citit un articol al lui Jeremy Shoemaker pe aceasta tema. El propunea folosirea unui fisier robots.txt, care este [...]

Very helpful post for me as I have been looking how to use the robots.txt file in this way for some time.

[...] Wordpress robots.txt tips against duplicate content Disallow: /*? - this blocks all urls with a ? in them. A good way to avoid duplicate content issues with wordpress blogs. Obviously you only want to use this if you have changed your url structure to not be 100% ?=.   [...]

I didn’t initially include a robots.txt in my blog and never had any issues with dupe content. It wasn’t until just recently I decided to add one, more for experimental purposes. So far, search engine traffic hasn’t improved or declined either way. Wordpress out of the box isn’t great for SEO purposes, but with minor tweeks, I find that a robots.txt isn’t really necessary.

-Guy
http://www.nullamatix.com

Thanks for the tips! I never thought of duplicate content with categories and archives before.

I have a question for anyone out there… if you want to have a blogroll on every page and post of your blog could you simply add nofollow tags to keep the links in the blogroll from being slapped? Or is it better to only have the blogroll on the home page?

[...] ist. Um dies wirkungsvoll zu vermeiden habe ich eine feine und vor allem schnelle Lösung bei Shoemoney.com gefunden. Er benutzt diese [...]

I noticed you have this:

Disallow: /index.php

in your robots.txt file. I wonder if that is a bad idea?

http://www.shoemoney.com/robots.txt

Ya,I dont use Disfollows..

Why my post cann’t be displayed.

[...] Shoemoney hat vor einigen Tagen darüber berichtet wie man mit ein paar Einträgen in der robots.txt solche doppler vermeidet. In diesem Fall ist die Liste mit Befehlen auf Wordpress angepasst, kann aber auch für andere Systeme genutzt werden (evtl. Anpassungen nötig). [...]

Thanks for this Jeremy. I have been looking for a good robots.txt file. I have no idea what to put in, so this will help.

[...] WordPress Robots.txt Tips Against Duplicate Content from Shoemoney. [...]

[...] Sollte man nun besser die oben angegebenen Plugins oder die robots.txt-Methode verwenden? Um die Unterschiede zu verstehen, muss man ein weniger tiefer in SEO-Welten abtauchen: während die beschriebenen Plugins die von Google vorgesehene Syntax noindex bzw. nofollow in den Header der betreffenden Dateien einfügen, sorgt die robots.txt-Variante dafür, dass überhaupt nie auf die betreffenden Seiten zugegriffen wird. Ob die beiden Varianten in der Praxis einen Unterschied machen, darüber streiten derzeit die SEO-Experten - siehe auch die Diskussion zum betreffenden Eintrag bei Shoemoney. [...]

Thats good that they added it, duplication is bad.

Great post with some great descriptions of what these certain words will “do.” Thanks for the post!

[...] are less likely to suffer from the penalties of duplicate content. WordPress users should see the article on Shoemoney about robots.txt [...]

[...] to basics” type post when he shared his tips on how to use the robots.txt in WordPress to prevent duplicate content. This is a great reference to use when editing your robots.txt to tweak your site and ensure you [...]

wow!!! Never knew that..??

Great tips, I’ll enhance my robots.txt file ASAP

[...] Als robots.txt speichern. Das wurde alles für Wordpress optimiert. [...]

[...] talks about his robots.txt file and how it guards against duplicate content in search engine results. Most of the strategies he’s using can be replicated for Movable Type and TypePad users. [...]

[...] likely to suffer from the penalties of duplicate content. WordPress users should see the article on Shoemoney about robots.txt [...]

Shoe, I just checked your actual robots.txt. Why do you have;

Disallow: /sitemap.xml

That seems like trouble?

[...] more tips on optimizing robots.txt for WordPress, check out Shoemoney’s suggestions. And keep in mind that like Shoemoney, I am not an SEO. I’ve just been using this method for [...]

[...] Line: We looked at Josh’s robots.txt post, as well as at ShoeMoney’s robot.txt post to figure out what we want our robots.txt file to look [...]