7

The pages should not be indexed if there are no direct links to them. Search engines don't guess at URLs, so I am guessing that a large part of your problem here is with your own code-- that is, you are generating links somewhere that the engines can follow. If it were me, I'd create a new post type for this usage and register it with 'public'->false so ...


6

You don't need to add robots.txt file to the root of your site. robots.txt file is generated in real time, when you visit http://mysite.com/robots.txt. The function, responsible for creation of this file, is do_robots. If you wish to add your own directives, just write your hook for robots_txt filter, like this: add_filter( 'robots_txt', '...


6

WordPress generate a dynamic robots.txt which does not physically exists. To remove/disable it you have two options: Option 1: Remove do_robots action in your theme functions.php or plugin remove_action('do_robots', 'do_robots'); The action do_robots is still available to be added again by other plugins. Option 2: Create a real robots.txt file, put it ...


4

Thought this was a great question so I went digging. In default-filters.php on line 208 there's add_action('wp_head', 'noindex', 1); as of WordPress 4.1. The noindex() function in turn checks to see if you have set blog_public option to 0. If you have, it calls wp_no_robots() which is simply: function wp_no_robots() { echo "<meta name='robots' ...


4

A lot of time since this quesiton and answer were posted. Since then things has changed a lot. The typical recommendation about disallow crawlers to access wp-content/themes, wp-content/plugins, wp-content/cache, wp-includes, and any other directory that contains CSS or js files needed in the site, are no longer valid. For example, lets talk about Google. ...


4

Well those are two different things: A redirect means they are not accessible any more at all. noindex just means that search engines ignore it while it is still accessible if you access the URL. So I'd recommend option 1. This is a simple way of doing this that you can improve an. (E.g. this expects to have a static front page set and doesn't handle any ...


3

Once I experienced the same issue, this is what I did to fix the issue. Edit the robots.txt file directly (using FTP/SSH), User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ There are two reasons if the robots files not updated when you edited using a plugin. File permission. Some other plugin is reverting the changes. Also try to update the ...


3

First of all, in order for Wordpress to generate a robots.txt for you you must be using a non-default permalink structure. Make sure you've selected an option in the Settings > Permalinks menu. Also, if a robots.txt file exists at your root directory it will override the setting in Wordpress. It looks like you already have a robots.txt file and that is the ...


2

Two options: Create a static file robots.txt. Highly recommended. Filter 'robots_txt': add_filter( 'robots_txt', 'wpse_77969_robots' ); function wpse_77969_robots() { status_header( 204 ); return ''; }


2

If it is because of SEO and the warnings in the Google Search console, these can be ignored. wp / Page2 and so on should still be indexed. The this answer and the article with the answer from google: For a while, SEOs thought it might be a good idea to add a noindex robots meta tag to page 2 and further of a paginated archive. This would prevent people from ...


2

Please go to Settings > Reading or wp-admin/options-reading.php If it's not unchecked it will not allow search engines to reach. For further modification information related to robots you can use the below plugin to modify robots.txt file: Plugin - Multipart robots.txt editor To see your robots.txt file you would visit the link as: http://www....


2

I suppose this ended up working for me. I was more hoping for some kind of better filter but it works just as well. Throw this in a functions.php file and you're good to go. /** No Index No Follow Entire Website **/ function nofollow_meta() { echo "<meta name='robots' content='noindex,nofollow' />\n"; } add_action( 'wp_head', 'nofollow_meta', 1 ); ...


2

I just tested the 'robots_txt' filter on a single installation to modify the output of the virtual /robots.txt that WordPress displays and it worked fine for me: add_filter('robots_txt', 'wpse_248124_robots_txt', 10, 2); function wpse_248124_robots_txt($output, $public) { return 'YOUR DESIRED OUTPUT'; } What is really happening when you try to reach /...


1

function noindex_comments_pages() { global $cpage; if (!empty($cpage) && $cpage > 1) { echo '<meta name="robots" content="noindex">'; echo "\n"; } } add_action( 'wp_head', 'noindex_comments_pages', 9 );


1

Here's the thing There are multiple ways to do what you want, from adding meta tags, to passing headers, but because, you tagged your question with robots.txt So i consider it off-topic to discuss any other solutions. Considering your demand you need to have this as your robots.txt This disallowed access to wp-admin, but depending on use case, you may need ...


1

Using robots.txt Yes you could use a robots.txt file for this, simply add the following into this file: User-agent: * Disallow: /wp-content/ Notice that you can have multiple Disallow directives if you would like to restrict indexing of other folders as well. User-agent: * Disallow: /wp-content/ Disallow: /wp-admin/ If you would like to allow indexing on a ...


1

You can safely place your robot.txt file as well as the sitemap.xml file into your websites root directory and NOT the WordPress directory. When a robot visits your site, the first place it checks is http://sitename.com/robots.txt, hence you want to keep these files at the root of your domain. You can also disallow what the robot has access to User-...


1

Okay, after digging into various areas, I noticed I have a security plugin that is blocking these changes. Problem solved by whitelisting, :) in case anyone else has this issue, check your security settings.


1

Just delete file robots.txt. WordPress is generating virtual robots.txt file, and Yoast SEO plugin (and other plugins too) allows you to customize it. But, if you have an actual robots.txt file, it will prevent virtual one to work.


1

Well, that message is pretty self explanatory. There is no sitemap linked in your robots.txt file. So you check if you have a sitemap (if not install a plugin that will generate one for you) and add this line to your robots.txt file in the root of your site: Sitemap: http://www.example.com/sitemap.xml


1

For those who are using WordPress as CMS for their site, you can bypass your web hosting server rules by simply removing your robots.txt file and instead modifying the virtual one generated by WordPress. You just have to add a filter to the functions.php file of your theme. Here's the code snippet: //* Append directives to the virtual robots.txt add_filter(...


1

Similar problem occurred to me, and I got the solution following these: Step 1: Take a backup of your .htaccess file and then remove it (Don't worry, on next refresh WordPress will create one for you) Step 2: If there's no robots.txt exists, create one with blank page Step 3: Resubmit the sitemap to google webmasters tools Step 4: Additionally use "Crawl as ...


1

Your robots file as of right now: User-agent: Googlebot Disallow: User-agent: * Disallow: Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-includes/ Sitemap: http://www.yadfaeq.com You aren't specifying anything for Googlebot so you should remove it. Additionally, Disallow: could be interpreted as Disallow: / which would block your entire site. ...


1

Normally, if there's a WordPress file on disk, that'll be served first directly by Apache or Nginx, before WordPress gets involved. This is done in your virtualhost config, e.g. in Nginx you'll typically find the following, which tells it to try actual files first before letting index.php handle the URL and generate a page on demand. location / { ...


1

It seems to be a WP default setting, as many Webmasters have gotten this warning and never edited the robots.txt. Removing all the disallows is the easiest solution, but I assume you want some or all of those directories blocked. Google is only concerned about the .js and .css files, so you could in theory edit the robots.txt to include: User-Agent: ...


1

From WordPress v3.5 this menu has moved to "Settings" -> "Reading" From here you can easily update the text of your robots.txt file. See https://wordpress.org/support/topic/wp-robots-txt-privacy-doesnt-appear-in-settings for further info


1

User-agent They are Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses. Disallow It tells the robots that it should not visit the pages or directory of ...


1

If you have an actual robots.txt file in the root of your site then WP/plugins will be unable to override it. Its related functionality works with "virtual" file, when actual file doesn't exist. Otherwise it might be plugin conflict, try one related plugin at a time.


1

You should follow Joost de Valk's current approach where very little is blocked in robots.txt, but also understand that each site will have a uniquely appropriate policy that will need to be reviewed and changed over time. Many of the answers given here previously are dated and will result in SEO self-sabotage since Google checks for "mobile friendliness" ...


1

As for robots.txt file. This will work with major search engines (Google, Bing/Yahoo, Yandex, Baidu). User-agent: * Disallow: *.php Disallow: *.js Disallow: *.inc Disallow: *.css Disallow: *.gz Disallow: *.wmv Disallow: *.cgi Disallow: *.xhtml Disallow: *.xlsx Disallow: *.doc Disallow: *.pdf Disallow: *.zip Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: ...


Only top voted, non community-wiki answers of a minimum length are eligible