Questions tagged [robots.txt]

This is a file that contains directions for search spiders. Its content depends on whether the user has allowed search engines to crawl the site.

Filter by
Sorted by
Tagged with
0 votes
0 answers
13 views

How to dissallow all wordpress posts in robots.txt?

Well, I have a question. How can I dissallow posts in the robots.txt file? My current robots.txt file: User-agent: Scrapy Allow: / User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php ...
  • 1
0 votes
0 answers
14 views

Hide plugins and theme from public

I like to hide all or some of my plugins and themes and CMS from public such that the users cannot detect which plugins/themes/CMS I used to build my website. Please let me know if it is possible to ...
  • 103
0 votes
1 answer
38 views

Use a filter for wp_robots to block CPT/feed/

We have a CPT (with base slug templates) that somehow included in every single post's head a link to /templates/post/feed/. I've removed this link to the feed from the head, but want to block robots ...
  • 1,588
0 votes
2 answers
1k views

Why does do_robots() Allow: /wp-admin/admin-ajax.php by default?

This is happening with a fresh, new WordPress install. But all you need to do is look at the crazy do_robots() function, which outputs the following... User-agent: * Disallow: /wp-admin/ Allow: /wp-...
  • 431
0 votes
0 answers
50 views

WordPress site using Yoast SEO being blocked from indexing by robots.txt

Morning all! I've just built a website for a client, using WordPress and the Yoast SEO plugin (free version). https://www.fly2help.org/ Whilst they're really happy with the website, I've noticed an ...
  • 101
0 votes
0 answers
35 views

How to stop google from indexing spammed internal search results?

Google is indexing all kinds of spammed internal search results from my wordpress site, and I can't figure out how to put a stop to it. I added Disallow: /search/ to my robots.txt file, but that didn'...
1 vote
1 answer
428 views

Set noindex page-comment from Pages 2, 3 and More?

I use All in one seo pack plugin. I see that Google indexed page comments (that are a pages generated from paginated comments). The example url are the followings: URL canonical - mydomain.com/url-...
  • 51
0 votes
3 answers
5k views

How to prevent Google from indexing the /wp-content/ directory?

I received an error from google saying that I have mobile usability errors on my site. the "page" that they are referring to is www.example.com/wp-content/plugins/ag-admin. I'm not asking ...
  • 340
1 vote
1 answer
115 views

Why does Google Image Search still display my images?

I am having trouble with a few image files that I would like to block for Google Image search. I am using Yoast SEO Plugin, which allows me to directly edit the robots.txt file. A few people with ...
0 votes
0 answers
141 views

Bing/msn bots is heavily requesting random of my website

I am facing a big problem with my server. I have a website that keeps getting massive page requests coming from "Bing/Msn" bot every second or two and the ip changes now and then. Which is putting a ...
0 votes
1 answer
1k views

How to noindex, follow a specific category wordpress?

Google appears to be ignoring my robots.txt (despite being valid) How do I noindex, follow a specific category without plugins?
0 votes
0 answers
520 views

Robots.txt file not updating

One of our website robots.txt file is not getting updated,deleted .htaccess and robots.txt file from root folder from one of the suggested answers Why does Google Webmaster Tools say my whole site is ...
  • 1
0 votes
1 answer
71 views

Why are some user agents still getting a meta robots tag with noindex from my site?

A friend pointed out to me that a website is not in Google because of the presence of a <meta name='robots' content='noindex,follow' /> tag in the home page. Ok, should be easy enough to turn ...
0 votes
1 answer
506 views

Where to put robot.txt and sitemap.xml for wordpress installed in its own directory

My website is setup where wordpress is installed in it's own subdirectory, but the content is served as though it's at the domain root. (This wordpress codex page details the configuration) Wordpress ...
0 votes
1 answer
45 views

Google saying is blocked by robots.txt without robots.txt on my website

Everything is in the title. I have no robots.txt on my website. I've enabled search engine indexing in the global settings. But still, Google cannot crawl my website due to an non-existing robots.txt. ...
  • 25
-1 votes
1 answer
23 views

How to view the code for a certain page?

Google is indexing pages that I used like a year ago and now they are 404. I want to add a noindex tag to them, but can't figure out how to open the code for a desired page so I can put in the tag. In ...
1 vote
1 answer
3k views

Disallow author with robots.txt

I would like to disallow the authors (of posts for example) in the file robots.txt. In fact, I noticed that for some of my sites, Google indexed the authors of articles, which I don't want to. Someone ...
0 votes
1 answer
703 views

How to remove Wordpress Virtual Robots.txt that keeps overriding my root robots.txt? [closed]

So here is the issue. I am trying to remove the virtual robots.txt file in a wordpress site. I realized this was generated with a plugin (xml sitemap plugin) prior to me working on the site. Here is ...
  • 44
1 vote
1 answer
44 views

install robots.txt to wordpress website admin

our website is http://syno-int.co.kr and using wordpress. I tried to install robots.txt to plugin section, but when I put zip file in it, it shows like this. Unable to create directory wp-content/...
  • 11
0 votes
1 answer
2k views

Old robots.txt file not changing, can't update to the current robots.txt

While my wordpress website was in production, I created a robots.txt file to disallow everything. When the site was ready, I deleted the robots.txt file through cpanel and never thought much about it. ...
1 vote
1 answer
1k views

Virtual robots.txt missing

on every WordPress installation there is a default virtual robots.txt file with the following User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php but on my 2 last project /robots.txt ...
-2 votes
2 answers
941 views

Google Webmaster Tools error "Url blocked by robots.txt."

I have a WordPress website, I submitted it to Google Webmaster and after submitting the sitemap, it's giving me error "URL blocked by robots.txt". Please help, what should I do?
  • 1
3 votes
1 answer
3k views

Generating robots.txt dynamically

I have a subfolder WP installation. It serves multiple domains, which are linked to the same folder. The requested domain is received in wp-config.php (from the $_SERVER variable) and used to define ...
2 votes
3 answers
7k views

"No sitemap linked in your robots.txt file"

Relatively new to anything that doesn't require common sense in WordPress. My previous site was built poorly etc, so cutting a long story short I installed a new theme and started building up pages ...
  • 21
0 votes
1 answer
80 views

Question about robots.txt file on WordPress site

I don't know why, but Google started indexing some pages that we don't have on our server. Mostly pages with the ?tag=22&page=30 query string. By the way, we don't have any "tag", and as result ...
  • 11
3 votes
2 answers
306 views

Correct way to hide pseudo pages from being shown?

The homepage on a site we've developed has four pieces of content: the parent page and three further "pages" of text which we bring into the parent page using PHP. Call them Foo, Bar1, Bar2, Bar3 if ...
  • 460
7 votes
4 answers
8k views

Robots.txt not updating

When I try to google search my web address, the follow comes up as the meta description "A description for this result is not available because of this site's robots.txt – learn more." The ...
-1 votes
1 answer
38 views

No exposure on search engines

I have a blog but it doesn't show on any search engine. Why can't my wordpress blog be seen but I am always publishing.
1 vote
1 answer
3k views

robots.txt Not Updating

Today I received an email from Webmaster Tools stating that "Googlebot cannot access CSS and JS files". So I set about updating my robots.txt file located in the root of my server. Original Contents: ...
0 votes
2 answers
2k views

Disallow: /wp-* in robots.txt?

I have just received an email from Google advising that it can't accessing certain javascript and css content from my site. I've looked at the robots.txt file which contains: User-agent: * ...
1 vote
1 answer
68 views

Where is the "Privacy" menu for the "WP Robots Txt" plugin?

I want to be able to update the robots.txt for my WordPress site. From brief research it looks like the "WP Robots Txt" plugin is the easiest. However the tutorials eg https://www.youtube.com/watch?v=...
  • 181
1 vote
0 answers
332 views

Robots.txt with WordPress and Joomla in same server

I have this configuration in my server root (hosted): root -> a Wordpress blog DirectoryJ -> a Joomla site DirectoryWP -> Another Wordpress blog I don't know how to handle the robots.txt. Joomla 3.4 ...
  • 11
0 votes
1 answer
2k views

What is the meaning of User-agent and Disallow in robots.txt file in wordpress

What is the meaning of User-agent and Disallow in robots.txt file in WordPress. I am little bit confused in this.
0 votes
1 answer
328 views

Trying to update robots.txt [closed]

I have installed and activaded the WP Robots txt plugin. Then I have gone to wp-admin/options-reading.php I have pasted a new robotst.txt content and Save it. But I can not see the new content at ...
  • 101
6 votes
2 answers
1k views

NoFollow Entire Website

By default whenever you disable indexing via Admin Settings [ x ] Discourage search engines from indexing this site It adds a meta tag in the header like so: <meta name='robots' content='...
  • 19.8k
4 votes
1 answer
959 views

Stuck on my server root folder, robots.txt file not deleting

I remove robots.txt file in my server root but google can't delete that. Webmastertool to create robotx.txt file How can I remove or disable.
  • 62
-1 votes
1 answer
2k views

Google is unable to crawl Robots.txt file of my website and please help me

Today I got a mail from Google webmasters, that Google bots are unable to find robots.txt file of my website. I also tried to locate the file by http://www.mydomain.com/robots.txt but the file isn't ...
3 votes
2 answers
5k views

Unwanted "crawl delay: 10" line added to my robots.txt

I've noticed that when I make a request to my robots.txt page, I get it with a crawl delay: 10 line added to it. I double-checked the file, and it doesn't have that line, so why does it appear? Is it ...
1 vote
3 answers
6k views

Why does Google Webmaster Tools say my whole site is blocked by robots.txt when I have no robots.txt?

I submitted a sitemap for my WordPress site to Google Webmaster Tools. Several times. I keep getting the same messages: "Url blocked by robots.txt." and "Sitemap contains urls which are blocked by ...
0 votes
1 answer
1k views

Disable Ajax for Spiders

We have a site on wpegnine that gets about 20k visits a day and heavily depends on ajax for our inbound marketing strategy (lead/visitor processing). These ajax calls are killing our server ...
  • 154
0 votes
1 answer
159 views

Hide pages from google and visitors

I have wordpress site where i have few custom type pages. I load the content of those pages with jquery load. But I dont want that those pages are directly visible. So if user goes to http://www....
1 vote
1 answer
3k views

How to solve Robots.txt problem with WordPress fresh install?

I installed a fresh copy of WordPress few weeks ago and I have been having some strange problems. 1- I get this message : "Comments for FirstName LastName name-last.com/‎ A description for this ...
0 votes
2 answers
3k views

Add robots.txt to root

I have a new Wordpress 3.5.1 install (hosted on Dreamhost FWIW) that I do not want to be indexed by search engines. I would like to serve a simple robots.txt with Disallow: / for all user agents. I ...
  • 117
2 votes
1 answer
4k views

How to stop WordPress showing a virtual Robots.txt?

I can see the robots.txt file here : http://persian.cc/robots.txt But Google can't ! I know that I can find robots.txt files in the root of a website, but hey there is no robots.txt file in mine ...
0 votes
2 answers
308 views

Is this robots.txt making Google not to show some of my content?

http://chineselearnonline.com/ User-agent: * Disallow: /feed/ Disallow: /trackback/ Disallow: /wp-admin/ Disallow: /wp-content/ Disallow: /wp-includes/ Disallow: /xmlrpc.php Disallow: /wp- Sitemap: ...
  • 3,717
0 votes
1 answer
145 views

wp_redirect and search engine bots

I have some pages and categories you have to be looged in to have access to. I am using wp_redirect to send anyone who is not logged in to the front page if they try to open those pages. Is it enough ...
  • 161
1 vote
1 answer
284 views

Worried I have a funky .htaccess for WP site in light of strange search engine behavior

My friend started a new (non-commercial) blog 5+ months ago, but Google + Bing are acting like we're telling it to not be indexed. I don't mean it ranks poorly. I mean it doesn't rank at all. I ...
  • 11
3 votes
2 answers
3k views

Create unique robots.txt for every site on multisite-installation

Does anyone know how I can have a unique robots.txt file for every domain served by my wp-multisite-installation? I did a search on plugins but couldn't find anything suitable.
  • 33
0 votes
2 answers
733 views

Can you please verify my robots.txt for wordpress

I am a newb in writing this and since this is dangerous stuff to fidlle with (if you are not sure what you are doing) can you please verify it? I use wordpress multisite, this is why I chose to allow ...
0 votes
1 answer
1k views

auto generated robots.txt problem

I have a site visibility in privacy settings set to allow search engines to see my site but for some reason when i go to the auto generated robots.txt file it shows User-agent: * Disallow: switching ...
  • 2,984