The robots tag has no wiki summary.
0
votes
0answers
15 views
Visitors occasionally see robots.txt instead of content
I'm running into a strange issue that I've never seen before: visitors to a Wordpress-powered website occasionally see the contents of Wordpress's automatically-generated robots.txt instead of the ...
-2
votes
1answer
86 views
WP in Subdirectory — Multiple robots.txt and .htaccess files?
Assume you have a static HTML site with its own robots.txt and sitemap.xml at www.mysite.com.
Now you add WordPress blog as an adjunct to the site at: www.mysite.com/blog
Do you need to have a ...
0
votes
2answers
351 views
Add robots.txt to root
I have a new Wordpress 3.5.1 install (hosted on Dreamhost FWIW) that I do not want to be indexed by search engines. I would like to serve a simple robots.txt with Disallow: / for all user agents.
I ...
0
votes
1answer
122 views
How to stop WordPress showing a virtual Robots.txt?
I can see the robots.txt file here :
http://persian.cc/robots.txt
But Google can't !
I know that I can find robots.txt files in the root of a website, but hey there is no robots.txt file in mine ...
0
votes
2answers
52 views
Is this robots.txt making Google not to show some of my content?
http://chineselearnonline.com/
User-agent: *
Disallow: /feed/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /xmlrpc.php
Disallow: /wp-
Sitemap: ...
0
votes
1answer
42 views
wp_redirect and search engine bots
I have some pages and categories you have to be looged in to have access to. I am using wp_redirect to send anyone who is not logged in to the front page if they try to open those pages. Is it enough ...
1
vote
1answer
95 views
Worried I have a funky .htaccess for WP site in light of strange search engine behavior
My friend started a new (non-commercial) blog 5+ months ago, but Google + Bing are acting like we're telling it to not be indexed.
I don't mean it ranks poorly. I mean it doesn't rank at all.
I ...
2
votes
1answer
167 views
Create unique robots.txt for every site on multisite-installation
Does anyone know how I can have a unique robots.txt file for every domain served by my wp-multisite-installation? I did a search on plugins but couldn't find anything suitable.
0
votes
1answer
139 views
Can you please verify my robots.txt for wordpress
I am a newb in writing this and since this is dangerous stuff to fidlle with (if you are not sure what you are doing) can you please verify it?
I use wordpress multisite, this is why I chose to allow ...
0
votes
1answer
123 views
auto generated robots.txt problem
I have a site visibility in privacy settings set to allow search engines to see my site but for some reason when i go to the auto generated robots.txt file it shows
User-agent: *
Disallow:
switching ...
1
vote
1answer
182 views
Questions about Robots.txt
I need some help with robots.txt.
Under Privacy Settings, I selected "I would like my blog to be visible to everyone, including search engines..."
As I understand it, WordPress automatically ...
0
votes
2answers
2k views
Setting up robots.txt with Yoast's Seo plugin
I was looking for a decent robots.txt solution for my wordpress sites and after some searching I found Yoast's SEO plugin that provides a complete SEO solution including indexation.
I set it up it to ...
5
votes
4answers
879 views
What is a good robots.txt?
What is the "best" setup for robots.txt?
I'm using the following permalink structure /%category%/%postname%/.
My robots.txt currently looks like this (copied from somewhere a long time ago):
...
1
vote
3answers
283 views
Preventing Search Engines Indexing Pages 2, 3 and More?
Do you know how to prevent indexing of pages past the home page in WP?
I mean I don't want mysite.com/page/2, mysite.com/page/3 to be indexed.
This is because I use home.php for my theme, so that ...
