Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [robots.txt]

This is a file that contains directions for search spiders. Its content depends on whether the user has allowed search engines to crawl the site.

7
votes
6answers
4k views

What is a good robots.txt?

What is the "best" setup for robots.txt? I'm using the following permalink structure /%category%/%postname%/. My robots.txt currently looks like this (copied from somewhere a long time ago): User-...
6
votes
2answers
944 views

NoFollow Entire Website

By default whenever you disable indexing via Admin Settings [ x ] Discourage search engines from indexing this site It adds a meta tag in the header like so: <meta name='robots' content='...
5
votes
3answers
4k views

Robots.txt not updating

When I try to google search my web address, the follow comes up as the meta description "A description for this result is not available because of this site's robots.txt – learn more." The ...
4
votes
1answer
698 views

Stuck on my server root folder, robots.txt file not deleting

I remove robots.txt file in my server root but google can't delete that. Webmastertool to create robotx.txt file How can I remove or disable.
3
votes
2answers
294 views

Correct way to hide pseudo pages from being shown?

The homepage on a site we've developed has four pieces of content: the parent page and three further "pages" of text which we bring into the parent page using PHP. Call them Foo, Bar1, Bar2, Bar3 if ...
3
votes
2answers
2k views

Create unique robots.txt for every site on multisite-installation

Does anyone know how I can have a unique robots.txt file for every domain served by my wp-multisite-installation? I did a search on plugins but couldn't find anything suitable.
3
votes
1answer
2k views

Generating robots.txt dynamically

I have a subfolder WP installation. It serves multiple domains, which are linked to the same folder. The requested domain is received in wp-config.php (from the $_SERVER variable) and used to define ...
3
votes
2answers
4k views

Unwanted “crawl delay: 10” line added to my robots.txt

I've noticed that when I make a request to my robots.txt page, I get it with a crawl delay: 10 line added to it. I double-checked the file, and it doesn't have that line, so why does it appear? Is it ...
1
vote
4answers
2k views

Preventing Search Engines Indexing Pages 2, 3 and More?

Do you know how to prevent indexing of pages past the home page in WP? I mean I don't want mysite.com/page/2, mysite.com/page/3 to be indexed. This is because I use home.php for my theme, so that ...
1
vote
1answer
263 views

Worried I have a funky .htaccess for WP site in light of strange search engine behavior

My friend started a new (non-commercial) blog 5+ months ago, but Google + Bing are acting like we're telling it to not be indexed. I don't mean it ranks poorly. I mean it doesn't rank at all. I ...
1
vote
1answer
3k views

“No sitemap linked in your robots.txt file”

Relatively new to anything that doesn't require common sense in WordPress. My previous site was built poorly etc, so cutting a long story short I installed a new theme and started building up pages ...
1
vote
1answer
2k views

robots.txt Not Updating

Today I received an email from Webmaster Tools stating that "Googlebot cannot access CSS and JS files". So I set about updating my robots.txt file located in the root of my server. Original Contents: ...
1
vote
1answer
63 views

Where is the “Privacy” menu for the “WP Robots Txt” plugin?

I want to be able to update the robots.txt for my WordPress site. From brief research it looks like the "WP Robots Txt" plugin is the easiest. However the tutorials eg https://www.youtube.com/watch?v=...
1
vote
3answers
6k views

Why does Google Webmaster Tools say my whole site is blocked by robots.txt when I have no robots.txt?

I submitted a sitemap for my WordPress site to Google Webmaster Tools. Several times. I keep getting the same messages: "Url blocked by robots.txt." and "Sitemap contains urls which are blocked by ...
1
vote
1answer
2k views

How to solve Robots.txt problem with WordPress fresh install?

I installed a fresh copy of WordPress few weeks ago and I have been having some strange problems. 1- I get this message : "Comments for FirstName LastName name-last.com/‎ A description for this ...
1
vote
1answer
657 views

Questions about Robots.txt

I need some help with robots.txt. Under Privacy Settings, I selected "I would like my blog to be visible to everyone, including search engines..." As I understand it, WordPress automatically ...
1
vote
1answer
39 views

install robots.txt to wordpress website admin

our website is http://syno-int.co.kr and using wordpress. I tried to install robots.txt to plugin section, but when I put zip file in it, it shows like this. Unable to create directory wp-content/...
1
vote
1answer
485 views

Virtual robots.txt missing

on every WordPress installation there is a default virtual robots.txt file with the following User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php but on my 2 last project /robots.txt ...
0
votes
2answers
2k views

Add robots.txt to root

I have a new Wordpress 3.5.1 install (hosted on Dreamhost FWIW) that I do not want to be indexed by search engines. I would like to serve a simple robots.txt with Disallow: / for all user agents. I ...
0
votes
1answer
3k views

How to stop WordPress showing a virtual Robots.txt?

I can see the robots.txt file here : http://persian.cc/robots.txt But Google can't ! I know that I can find robots.txt files in the root of a website, but hey there is no robots.txt file in mine ...
0
votes
2answers
306 views

Is this robots.txt making Google not to show some of my content?

http://chineselearnonline.com/ User-agent: * Disallow: /feed/ Disallow: /trackback/ Disallow: /wp-admin/ Disallow: /wp-content/ Disallow: /wp-includes/ Disallow: /xmlrpc.php Disallow: /wp- Sitemap: ...
0
votes
2answers
650 views

Can you please verify my robots.txt for wordpress

I am a newb in writing this and since this is dangerous stuff to fidlle with (if you are not sure what you are doing) can you please verify it? I use wordpress multisite, this is why I chose to allow ...
0
votes
1answer
37 views

Where to put robot.txt and sitemap.xml for wordpress installed in its own directory

My website is setup where wordpress is installed in it's own subdirectory, but the content is served as though it's at the domain root. (This wordpress codex page details the configuration) Wordpress ...
0
votes
1answer
133 views

How to remove Wordpress Virtual Robots.txt that keeps overriding my root robots.txt? [closed]

So here is the issue. I am trying to remove the virtual robots.txt file in a wordpress site. I realized this was generated with a plugin (xml sitemap plugin) prior to me working on the site. Here is ...
0
votes
1answer
1k views

Old robots.txt file not changing, can't update to the current robots.txt

While my wordpress website was in production, I created a robots.txt file to disallow everything. When the site was ready, I deleted the robots.txt file through cpanel and never thought much about it. ...
0
votes
2answers
1k views

Disallow: /wp-* in robots.txt?

I have just received an email from Google advising that it can't accessing certain javascript and css content from my site. I've looked at the robots.txt file which contains: User-agent: * ...
0
votes
1answer
991 views

What is the meaning of User-agent and Disallow in robots.txt file in wordpress

What is the meaning of User-agent and Disallow in robots.txt file in WordPress. I am little bit confused in this.
0
votes
1answer
223 views

Trying to update robots.txt [closed]

I have installed and activaded the WP Robots txt plugin. Then I have gone to wp-admin/options-reading.php I have pasted a new robotst.txt content and Save it. But I can not see the new content at ...
0
votes
1answer
936 views

auto generated robots.txt problem

I have a site visibility in privacy settings set to allow search engines to see my site but for some reason when i go to the auto generated robots.txt file it shows User-agent: * Disallow: switching ...
0
votes
3answers
4k views

Setting up robots.txt with Yoast's Seo plugin

I was looking for a decent robots.txt solution for my wordpress sites and after some searching I found Yoast's SEO plugin that provides a complete SEO solution including indexation. I set it up it to ...
0
votes
0answers
47 views

Robots.txt file not updating

One of our website robots.txt file is not getting updated,deleted .htaccess and robots.txt file from root folder from one of the suggested answers Why does Google Webmaster Tools say my whole site is ...
0
votes
1answer
21 views

Why are some user agents still getting a meta robots tag with noindex from my site?

A friend pointed out to me that a website is not in Google because of the presence of a <meta name='robots' content='noindex,follow' /> tag in the home page. Ok, should be easy enough to turn ...
0
votes
1answer
24 views

Google saying is blocked by robots.txt without robots.txt on my website

Everything is in the title. I have no robots.txt on my website. I've enabled search engine indexing in the global settings. But still, Google cannot crawl my website due to an non-existing robots.txt. ...
0
votes
1answer
607 views

Disallow author with robots.txt

I would like to disallow the authors (of articles for example) in the file robots.txt. In fact, I noticed that for some of my sites, Google indexed the authors of articles, which I don't want to. ...
0
votes
0answers
23 views

New Index coverage related issue for all the “Tag” pages in Google Webmaster tool

I have a blogging website in Wordpress. I am using Yoast SEO for sitemaps. I got the following error message in Google webmaster tool: Search Console has identified that your site is affected by 1 ...
0
votes
1answer
48 views

Question about robots.txt file on WordPress site

I don't know why, but Google started indexing some pages that we don't have on our server. Mostly pages with the ?tag=22&page=30 query string. By the way, we don't have any "tag", and as result ...
0
votes
0answers
289 views

Robots.txt with WordPress and Joomla in same server

I have this configuration in my server root (hosted): root -> a Wordpress blog DirectoryJ -> a Joomla site DirectoryWP -> Another Wordpress blog I don't know how to handle the robots.txt. Joomla 3.4 ...
0
votes
1answer
126 views

Hide pages from google and visitors

I have wordpress site where i have few custom type pages. I load the content of those pages with jquery load. But I dont want that those pages are directly visible. So if user goes to http://www....
0
votes
1answer
136 views

wp_redirect and search engine bots

I have some pages and categories you have to be looged in to have access to. I am using wp_redirect to send anyone who is not logged in to the front page if they try to open those pages. Is it enough ...
0
votes
1answer
1k views

Disable Ajax for Spiders

We have a site on wpegnine that gets about 20k visits a day and heavily depends on ajax for our inbound marketing strategy (lead/visitor processing). These ajax calls are killing our server ...
-1
votes
1answer
36 views

No exposure on search engines

I have a blog but it doesn't show on any search engine. Why can't my wordpress blog be seen but I am always publishing.
-1
votes
1answer
22 views

How to view the code for a certain page?

Google is indexing pages that I used like a year ago and now they are 404. I want to add a noindex tag to them, but can't figure out how to open the code for a desired page so I can put in the tag. In ...
-1
votes
1answer
1k views

Google is unable to crawl Robots.txt file of my website and please help me

Today I got a mail from Google webmasters, that Google bots are unable to find robots.txt file of my website. I also tried to locate the file by http://www.mydomain.com/robots.txt but the file isn't ...
-2
votes
2answers
859 views

Google Webmaster Tools error “Url blocked by robots.txt.”

I have a WordPress website, I submitted it to Google Webmaster and after submitting the sitemap, it's giving me error "URL blocked by robots.txt". Please help, what should I do?