The tag has no wiki summary.

learn more… | top users | synonyms (1)

0
votes
1answer
29 views

robots.txt Not Updating

Today I received an email from Webmaster Tools stating that "Googlebot cannot access CSS and JS files". So I set about updating my robots.txt file located in the root of my server. Original Contents: ...
0
votes
2answers
57 views

Disallow: /wp-* in robots.txt?

I have just received an email from Google advising that it can't accessing certain javascript and css content from my site. I've looked at the robots.txt file which contains: User-agent: * ...
1
vote
1answer
27 views

Where is the “Privacy” menu for the “WP Robots Txt” plugin?

I want to be able to update the robots.txt for my WordPress site. From brief research it looks like the "WP Robots Txt" plugin is the easiest. However the tutorials eg ...
0
votes
0answers
64 views

Robots.txt with WordPress and Joomla in same server

I have this configuration in my server root (hosted): root -> a Wordpress blog DirectoryJ -> a Joomla site DirectoryWP -> Another Wordpress blog I don't know how to handle the robots.txt. Joomla ...
0
votes
1answer
66 views

Trying to update robots.txt [closed]

I have installed and activaded the WP Robots txt plugin. Then I have gone to wp-admin/options-reading.php I have pasted a new robotst.txt content and Save it. But I can not see the new content at ...
5
votes
2answers
140 views

NoFollow Entire Website

By default whenever you disable indexing via Admin Settings [ x ] Discourage search engines from indexing this site It adds a meta tag in the header like so: <meta name='robots' ...
4
votes
1answer
414 views

Stuck on my server root folder, robots.txt file not deleting

I remove robots.txt file in my server root but google can't delete that. Webmastertool to create robotx.txt file How can I remove or disable.
0
votes
1answer
344 views

Google is unable to crawl Robots.txt file of my website and please help me

Today I got a mail from Google webmasters, that Google bots are unable to find robots.txt file of my website. I also tried to locate the file by http://www.mydomain.com/robots.txt but the file isn't ...
2
votes
0answers
641 views

Unwanted “crawl delay: 10” line added to my robots.txt

I've noticed that when I make a request to my robots.txt page, I get it with a crawl delay: 10 line added to it. I double-checked the file, and it doesn't have that line, so why does it appear? Is it ...
1
vote
3answers
2k views

Why does Google Webmaster Tools say my whole site is blocked by robots.txt when I have no robots.txt?

I submitted a sitemap for my WordPress site to Google Webmaster Tools. Several times. I keep getting the same messages: "Url blocked by robots.txt." and "Sitemap contains urls which are blocked by ...
0
votes
1answer
290 views

Disable Ajax for Spiders

We have a site on wpegnine that gets about 20k visits a day and heavily depends on ajax for our inbound marketing strategy (lead/visitor processing). These ajax calls are killing our server ...
0
votes
1answer
72 views

Hide pages from google and visitors

I have wordpress site where i have few custom type pages. I load the content of those pages with jquery load. But I dont want that those pages are directly visible. So if user goes to ...
1
vote
1answer
709 views

How to solve Robots.txt problem with WordPress fresh install?

I installed a fresh copy of WordPress few weeks ago and I have been having some strange problems. 1- I get this message : "Comments for FirstName LastName name-last.com/‎ A description for this ...
-2
votes
1answer
440 views

WP in Subdirectory — Multiple robots.txt and .htaccess files?

Assume you have a static HTML site with its own robots.txt and sitemap.xml at www.mysite.com. Now you add WordPress blog as an adjunct to the site at: www.mysite.com/blog Do you need to have a ...
0
votes
2answers
2k views

Add robots.txt to root

I have a new Wordpress 3.5.1 install (hosted on Dreamhost FWIW) that I do not want to be indexed by search engines. I would like to serve a simple robots.txt with Disallow: / for all user agents. I ...
0
votes
1answer
985 views

How to stop WordPress showing a virtual Robots.txt?

I can see the robots.txt file here : http://persian.cc/robots.txt But Google can't ! I know that I can find robots.txt files in the root of a website, but hey there is no robots.txt file in mine ...
0
votes
2answers
280 views

Is this robots.txt making Google not to show some of my content?

http://chineselearnonline.com/ User-agent: * Disallow: /feed/ Disallow: /trackback/ Disallow: /wp-admin/ Disallow: /wp-content/ Disallow: /wp-includes/ Disallow: /xmlrpc.php Disallow: /wp- Sitemap: ...
0
votes
1answer
76 views

wp_redirect and search engine bots

I have some pages and categories you have to be looged in to have access to. I am using wp_redirect to send anyone who is not logged in to the front page if they try to open those pages. Is it enough ...
1
vote
1answer
140 views

Worried I have a funky .htaccess for WP site in light of strange search engine behavior

My friend started a new (non-commercial) blog 5+ months ago, but Google + Bing are acting like we're telling it to not be indexed. I don't mean it ranks poorly. I mean it doesn't rank at all. I ...
2
votes
1answer
805 views

Create unique robots.txt for every site on multisite-installation

Does anyone know how I can have a unique robots.txt file for every domain served by my wp-multisite-installation? I did a search on plugins but couldn't find anything suitable.
0
votes
2answers
343 views

Can you please verify my robots.txt for wordpress

I am a newb in writing this and since this is dangerous stuff to fidlle with (if you are not sure what you are doing) can you please verify it? I use wordpress multisite, this is why I chose to allow ...
0
votes
1answer
228 views

auto generated robots.txt problem

I have a site visibility in privacy settings set to allow search engines to see my site but for some reason when i go to the auto generated robots.txt file it shows User-agent: * Disallow: switching ...
1
vote
1answer
524 views

Questions about Robots.txt

I need some help with robots.txt. Under Privacy Settings, I selected "I would like my blog to be visible to everyone, including search engines..." As I understand it, WordPress automatically ...
0
votes
3answers
4k views

Setting up robots.txt with Yoast's Seo plugin

I was looking for a decent robots.txt solution for my wordpress sites and after some searching I found Yoast's SEO plugin that provides a complete SEO solution including indexation. I set it up it to ...
6
votes
6answers
2k views

What is a good robots.txt?

What is the "best" setup for robots.txt? I'm using the following permalink structure /%category%/%postname%/. My robots.txt currently looks like this (copied from somewhere a long time ago): ...
1
vote
3answers
426 views

Preventing Search Engines Indexing Pages 2, 3 and More?

Do you know how to prevent indexing of pages past the home page in WP? I mean I don't want mysite.com/page/2, mysite.com/page/3 to be indexed. This is because I use home.php for my theme, so that ...