Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
7

The pages should not be indexed if there are no direct links to them. Search engines don't guess at URLs, so I am guessing that a large part of your problem here is with your own code-- that is, you are generating links somewhere that the engines can follow. If it were me, I'd create a new post type for this usage and register it with 'public'->false so ...


6

WordPress generate a dynamic robots.txt which does not physically exists. To remove/disable it you have two options: Option 1: Remove do_robots action in your theme functions.php or plugin remove_action('do_robots', 'do_robots'); The action do_robots is still available to be added again by other plugins. Option 2: Create a real robots.txt file, put it ...


6

You don't need to add robots.txt file to the root of your site. robots.txt file is generated in real time, when you visit http://mysite.com/robots.txt. The function, responsible for creation of this file, is do_robots. If you wish to add your own directives, just write your hook for robots_txt filter, like this: add_filter( 'robots_txt', '...


4

A lot of time since this quesiton and answer were posted. Since then things has changed a lot. The typical recommendation about disallow crawlers to access wp-content/themes, wp-content/plugins, wp-content/cache, wp-includes, and any other directory that contains CSS or js files needed in the site, are no longer valid. For example, lets talk about Google. ...


4

Well those are two different things: A redirect means they are not accessible any more at all. noindex just means that search engines ignore it while it is still accessible if you access the URL. So I'd recommend option 1. This is a simple way of doing this that you can improve an. (E.g. this expects to have a static front page set and doesn't handle any ...


4

Thought this was a great question so I went digging. In default-filters.php on line 208 there's add_action('wp_head', 'noindex', 1); as of WordPress 4.1. The noindex() function in turn checks to see if you have set blog_public option to 0. If you have, it calls wp_no_robots() which is simply: function wp_no_robots() { echo "<meta name='robots' ...


3

First of all, in order for Wordpress to generate a robots.txt for you you must be using a non-default permalink structure. Make sure you've selected an option in the Settings > Permalinks menu. Also, if a robots.txt file exists at your root directory it will override the setting in Wordpress. It looks like you already have a robots.txt file and that is the ...


3

Straight from the source, (line 1845 wp-includes/functions.php, 3.3.1): function do_robots() { header( 'Content-Type: text/plain; charset=utf-8' ); do_action( 'do_robotstxt' ); $output = "User-agent: *\n"; $public = get_option( 'blog_public' ); if ( '0' == $public ) { $output .= "Disallow: /\n"; } else { $site_url = ...


2

With a little bit of help, this is now mines (not to much different from everyone elses, apparently) User-agent: * Allow: / Disallow: /wp-content/ Disallow: /wp-admin/ Disallow: /cat/ Disallow: /key/ Disallow: /*? Disallow: /*.js$ Disallow: /*.inc$ Disallow: /*.css$ Disallow: /cgi-bin Disallow: /wp-admin Disallow:...


2

Two options: Create a static file robots.txt. Highly recommended. Filter 'robots_txt': add_filter( 'robots_txt', 'wpse_77969_robots' ); function wpse_77969_robots() { status_header( 204 ); return ''; }


2

If you changed your site path, the urls previously indexed by search engines are no longer valid. Google Webmaster Tools has a change of address tool: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=83106 In general, you need to : setup 301 redirects for your old urls create a new sitemap be patient


2

WordPress responds to requests to a robots.txt with dynamic content if such a file does not exist. That’s one way how the settings from wp-admin/options-privacy.php are used. I recommend to create a static robots.txt, just to make sure no plugin is getting in your way. Sample robots.txt User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-...


2

Please go to Settings > Reading or wp-admin/options-reading.php If it's not unchecked it will not allow search engines to reach. For further modification information related to robots you can use the below plugin to modify robots.txt file: Plugin - Multipart robots.txt editor To see your robots.txt file you would visit the link as: http://www....


2

I suppose this ended up working for me. I was more hoping for some kind of better filter but it works just as well. Throw this in a functions.php file and you're good to go. /** No Index No Follow Entire Website **/ function nofollow_meta() { echo "<meta name='robots' content='noindex,nofollow' />\n"; } add_action( 'wp_head', 'nofollow_meta', 1 ); ...


2

I just tested the 'robots_txt' filter on a single installation to modify the output of the virtual /robots.txt that WordPress displays and it worked fine for me: add_filter('robots_txt', 'wpse_248124_robots_txt', 10, 2); function wpse_248124_robots_txt($output, $public) { return 'YOUR DESIRED OUTPUT'; } What is really happening when you try to reach /...


1

Well, that message is pretty self explanatory. There is no sitemap linked in your robots.txt file. So you check if you have a sitemap (if not install a plugin that will generate one for you) and add this line to your robots.txt file in the root of your site: Sitemap: http://www.example.com/sitemap.xml


1

Once I experienced the same issue, this is what I did to fix the issue. Edit the robots.txt file directly (using FTP/SSH), User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ There are two reasons if the robots files not updated when you edited using a plugin. File permission. Some other plugin is reverting the changes. Also try to update the ...


1

Normally, if there's a WordPress file on disk, that'll be served first directly by Apache or Nginx, before WordPress gets involved. This is done in your virtualhost config, e.g. in Nginx you'll typically find the following, which tells it to try actual files first before letting index.php handle the URL and generate a page on demand. location / { ...


1

It seems to be a WP default setting, as many Webmasters have gotten this warning and never edited the robots.txt. Removing all the disallows is the easiest solution, but I assume you want some or all of those directories blocked. Google is only concerned about the .js and .css files, so you could in theory edit the robots.txt to include: User-Agent: ...


1

From WordPress v3.5 this menu has moved to "Settings" -> "Reading" From here you can easily update the text of your robots.txt file. See https://wordpress.org/support/topic/wp-robots-txt-privacy-doesnt-appear-in-settings for further info


1

User-agent They are Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses. Disallow It tells the robots that it should not visit the pages or directory of ...


1

If you have an actual robots.txt file in the root of your site then WP/plugins will be unable to override it. Its related functionality works with "virtual" file, when actual file doesn't exist. Otherwise it might be plugin conflict, try one related plugin at a time.


1

For those who are using WordPress as CMS for their site, you can bypass your web hosting server rules by simply removing your robots.txt file and instead modifying the virtual one generated by WordPress. You just have to add a filter to the functions.php file of your theme. Here's the code snippet: //* Append directives to the virtual robots.txt add_filter(...


1

Similar problem occurred to me, and I got the solution following these: Step 1: Take a backup of your .htaccess file and then remove it (Don't worry, on next refresh WordPress will create one for you) Step 2: If there's no robots.txt exists, create one with blank page Step 3: Resubmit the sitemap to google webmasters tools Step 4: Additionally use "Crawl as ...


1

Your robots file as of right now: User-agent: Googlebot Disallow: User-agent: * Disallow: Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-includes/ Sitemap: http://www.yadfaeq.com You aren't specifying anything for Googlebot so you should remove it. Additionally, Disallow: could be interpreted as Disallow: / which would block your entire site. ...


1

As for robots.txt file. This will work with major search engines (Google, Bing/Yahoo, Yandex, Baidu). User-agent: * Disallow: *.php Disallow: *.js Disallow: *.inc Disallow: *.css Disallow: *.gz Disallow: *.wmv Disallow: *.cgi Disallow: *.xhtml Disallow: *.xlsx Disallow: *.doc Disallow: *.pdf Disallow: *.zip Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: ...


1

The file robots.txt doesn't exclude all the search engines to crawl your site... Most search engines crawl the hole site without taking the robots.txt file in consideration. I think that the best way is to use meta tags in the theme of the blogs. Use the next meta tags for post and categories template files: <meta name="robots" content="index, follow" ...


1

Disallow with no trailing slash is the same as "allow all" - see here - nice find!


1

you can just make a file in your root named robots.txt if you don't already have one User-agent: * Disallow: /myPage is the format for that one page there is no allow, only disallow. disllowing pages you follow the above format WP may not have generated a robots.txt, I believe it only would have if you wanted your site not indexed I'm not sure if WP has ...


1

Writing a robots.txt is an easy process. Follow these simple steps: Open Notepad, Microsoft Word or any text editor and save the file as 'robots,' all lowercase, making sure to choose .txt as the file type extension (in Word, choose 'Plain Text' ). Next, add the following two lines of text to your file: User-agent: * Disallow: 'User-agent' is another ...


Only top voted, non community-wiki answers of a minimum length are eligible