The tag has no wiki summary.

learn more… | top users | synonyms (1)

0
votes
1answer
255 views

I'm using All in One SEO Pack plugin and can't find robots.txt in my root directory.

I found the problem that Google crawler can't crawl my WordPress site contents. To fix it, I tried to follow Google's instruction which add 2 lines in robots.txt file. But i can't find the file. I'm ...
1
vote
3answers
226 views

Why does Google Webmaster Tools say my whole site is blocked by robots.txt when I have no robots.txt?

I submitted a sitemap for my WordPress site to Google Webmaster Tools. Several times. I keep getting the same messages: "Url blocked by robots.txt." and "Sitemap contains urls which are blocked by ...
0
votes
1answer
61 views

Disable Ajax for Spiders

We have a site on wpegnine that gets about 20k visits a day and heavily depends on ajax for our inbound marketing strategy (lead/visitor processing). These ajax calls are killing our server ...
0
votes
1answer
60 views

Hide pages from google and visitors

I have wordpress site where i have few custom type pages. I load the content of those pages with jquery load. But I dont want that those pages are directly visible. So if user goes to ...
0
votes
1answer
226 views

How to solve Robots.txt problem with WordPress fresh install?

I installed a fresh copy of WordPress few weeks ago and I have been having some strange problems. 1- I get this message : "Comments for Yad Faeq yadfaeq.com/‎ A description for this result is not ...
0
votes
2answers
286 views

Can you please verify my robots.txt for wordpress

I am a newb in writing this and since this is dangerous stuff to fidlle with (if you are not sure what you are doing) can you please verify it? I use wordpress multisite, this is why I chose to allow ...
-2
votes
1answer
279 views

WP in Subdirectory — Multiple robots.txt and .htaccess files?

Assume you have a static HTML site with its own robots.txt and sitemap.xml at www.mysite.com. Now you add WordPress blog as an adjunct to the site at: www.mysite.com/blog Do you need to have a ...
5
votes
4answers
1k views

What is a good robots.txt?

What is the "best" setup for robots.txt? I'm using the following permalink structure /%category%/%postname%/. My robots.txt currently looks like this (copied from somewhere a long time ago): ...
0
votes
2answers
1k views

Add robots.txt to root

I have a new Wordpress 3.5.1 install (hosted on Dreamhost FWIW) that I do not want to be indexed by search engines. I would like to serve a simple robots.txt with Disallow: / for all user agents. I ...
0
votes
1answer
637 views

How to stop WordPress showing a virtual Robots.txt?

I can see the robots.txt file here : http://persian.cc/robots.txt But Google can't ! I know that I can find robots.txt files in the root of a website, but hey there is no robots.txt file in mine ...
0
votes
2answers
245 views

Is this robots.txt making Google not to show some of my content?

http://chineselearnonline.com/ User-agent: * Disallow: /feed/ Disallow: /trackback/ Disallow: /wp-admin/ Disallow: /wp-content/ Disallow: /wp-includes/ Disallow: /xmlrpc.php Disallow: /wp- Sitemap: ...
0
votes
1answer
64 views

wp_redirect and search engine bots

I have some pages and categories you have to be looged in to have access to. I am using wp_redirect to send anyone who is not logged in to the front page if they try to open those pages. Is it enough ...
1
vote
1answer
115 views

Worried I have a funky .htaccess for WP site in light of strange search engine behavior

My friend started a new (non-commercial) blog 5+ months ago, but Google + Bing are acting like we're telling it to not be indexed. I don't mean it ranks poorly. I mean it doesn't rank at all. I ...
2
votes
1answer
431 views

Create unique robots.txt for every site on multisite-installation

Does anyone know how I can have a unique robots.txt file for every domain served by my wp-multisite-installation? I did a search on plugins but couldn't find anything suitable.
0
votes
2answers
3k views

Setting up robots.txt with Yoast's Seo plugin

I was looking for a decent robots.txt solution for my wordpress sites and after some searching I found Yoast's SEO plugin that provides a complete SEO solution including indexation. I set it up it to ...
0
votes
1answer
163 views

auto generated robots.txt problem

I have a site visibility in privacy settings set to allow search engines to see my site but for some reason when i go to the auto generated robots.txt file it shows User-agent: * Disallow: switching ...
1
vote
1answer
364 views

Questions about Robots.txt

I need some help with robots.txt. Under Privacy Settings, I selected "I would like my blog to be visible to everyone, including search engines..." As I understand it, WordPress automatically ...
1
vote
3answers
362 views

Preventing Search Engines Indexing Pages 2, 3 and More?

Do you know how to prevent indexing of pages past the home page in WP? I mean I don't want mysite.com/page/2, mysite.com/page/3 to be indexed. This is because I use home.php for my theme, so that ...