The tag has no wiki summary.

learn more… | top users | synonyms (1)

-5
votes
0answers
13 views

surprising visitors on my wordpress site? [closed]

Hi I have a blog in wordpress http://www.programming-tutor.net/ where I plan to put some c programming tutorials never got time to finish it actually. but still site is live. Yesterday I received an ...
0
votes
1answer
117 views

Google is unable to crawl Robots.txt file of my website and please help me

Today I got a mail from Google webmasters, that Google bots are unable to find robots.txt file of my website. I also tried to locate the file by http://www.mydomain.com/robots.txt but the file isn't ...
0
votes
0answers
163 views

Unwanted “crawl delay: 10” line added to my robots.txt

I've noticed that when I make a request to my robots.txt page, I get it with a crawl delay: 10 line added to it. I double-checked the file, and it doesn't have that line, so why does it appear? Is it ...
1
vote
3answers
992 views

Why does Google Webmaster Tools say my whole site is blocked by robots.txt when I have no robots.txt?

I submitted a sitemap for my WordPress site to Google Webmaster Tools. Several times. I keep getting the same messages: "Url blocked by robots.txt." and "Sitemap contains urls which are blocked by ...
0
votes
1answer
144 views

Disable Ajax for Spiders

We have a site on wpegnine that gets about 20k visits a day and heavily depends on ajax for our inbound marketing strategy (lead/visitor processing). These ajax calls are killing our server ...
0
votes
1answer
70 views

Hide pages from google and visitors

I have wordpress site where i have few custom type pages. I load the content of those pages with jquery load. But I dont want that those pages are directly visible. So if user goes to ...
1
vote
1answer
377 views

How to solve Robots.txt problem with WordPress fresh install?

I installed a fresh copy of WordPress few weeks ago and I have been having some strange problems. 1- I get this message : "Comments for Yad Faeq yadfaeq.com/‎ A description for this result is not ...
-2
votes
1answer
339 views

WP in Subdirectory — Multiple robots.txt and .htaccess files?

Assume you have a static HTML site with its own robots.txt and sitemap.xml at www.mysite.com. Now you add WordPress blog as an adjunct to the site at: www.mysite.com/blog Do you need to have a ...
0
votes
2answers
1k views

Add robots.txt to root

I have a new Wordpress 3.5.1 install (hosted on Dreamhost FWIW) that I do not want to be indexed by search engines. I would like to serve a simple robots.txt with Disallow: / for all user agents. I ...
0
votes
1answer
741 views

How to stop WordPress showing a virtual Robots.txt?

I can see the robots.txt file here : http://persian.cc/robots.txt But Google can't ! I know that I can find robots.txt files in the root of a website, but hey there is no robots.txt file in mine ...
0
votes
2answers
260 views

Is this robots.txt making Google not to show some of my content?

http://chineselearnonline.com/ User-agent: * Disallow: /feed/ Disallow: /trackback/ Disallow: /wp-admin/ Disallow: /wp-content/ Disallow: /wp-includes/ Disallow: /xmlrpc.php Disallow: /wp- Sitemap: ...
0
votes
1answer
70 views

wp_redirect and search engine bots

I have some pages and categories you have to be looged in to have access to. I am using wp_redirect to send anyone who is not logged in to the front page if they try to open those pages. Is it enough ...
1
vote
1answer
121 views

Worried I have a funky .htaccess for WP site in light of strange search engine behavior

My friend started a new (non-commercial) blog 5+ months ago, but Google + Bing are acting like we're telling it to not be indexed. I don't mean it ranks poorly. I mean it doesn't rank at all. I ...
2
votes
1answer
550 views

Create unique robots.txt for every site on multisite-installation

Does anyone know how I can have a unique robots.txt file for every domain served by my wp-multisite-installation? I did a search on plugins but couldn't find anything suitable.
0
votes
2answers
312 views

Can you please verify my robots.txt for wordpress

I am a newb in writing this and since this is dangerous stuff to fidlle with (if you are not sure what you are doing) can you please verify it? I use wordpress multisite, this is why I chose to allow ...
0
votes
1answer
179 views

auto generated robots.txt problem

I have a site visibility in privacy settings set to allow search engines to see my site but for some reason when i go to the auto generated robots.txt file it shows User-agent: * Disallow: switching ...
1
vote
1answer
420 views

Questions about Robots.txt

I need some help with robots.txt. Under Privacy Settings, I selected "I would like my blog to be visible to everyone, including search engines..." As I understand it, WordPress automatically ...
0
votes
2answers
3k views

Setting up robots.txt with Yoast's Seo plugin

I was looking for a decent robots.txt solution for my wordpress sites and after some searching I found Yoast's SEO plugin that provides a complete SEO solution including indexation. I set it up it to ...
5
votes
4answers
1k views

What is a good robots.txt?

What is the "best" setup for robots.txt? I'm using the following permalink structure /%category%/%postname%/. My robots.txt currently looks like this (copied from somewhere a long time ago): ...
1
vote
3answers
385 views

Preventing Search Engines Indexing Pages 2, 3 and More?

Do you know how to prevent indexing of pages past the home page in WP? I mean I don't want mysite.com/page/2, mysite.com/page/3 to be indexed. This is because I use home.php for my theme, so that ...