Questions tagged [robots.txt]

This is a file that contains directions for search spiders. Its content depends on whether the user has allowed search engines to crawl the site.

Filter by
Sorted by
Tagged with
0 votes
2 answers
54 views

Wordpress does not generate robots.txt

My Wordpress does not generate a robots.txt file. I have no idea how to solve this and I hope somebody has met with the same problem and knows of a solution. Thanks in advance! Bye Ward
Ward van den Berg's user avatar
0 votes
2 answers
61 views

Can I remove these lines form my WordPress robots.txt

Can I remove this lines form my robots.txt when using WordPress: User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://domain/sitemap_index.xml -> I want to remove ...
AYNTK's user avatar
  • 7
1 vote
1 answer
62 views

Displaying the Virtural robots.txt file

The virtual robots.txt file is generated via the do_robots() function, along with any directives added via the robots_txt filter. But I can't figure out a way to display the generated robots.txt ...
Rick Hellewell's user avatar
-1 votes
1 answer
16 views

Want to block all posts of spacific author via robots.txt file

I am using WordPress, and we never show the author name in the URL of the article URL is domain.com/this-is-a-great-post but the author's name is visible in the article only the URL is without author'...
Sikander's user avatar
0 votes
0 answers
156 views

Hide plugins and theme from public

I like to hide all or some of my plugins and themes and CMS from public such that the users cannot detect which plugins/themes/CMS I used to build my website. Please let me know if it is possible to ...
asad's user avatar
  • 103
0 votes
1 answer
220 views

Use a filter for wp_robots to block CPT/feed/

We have a CPT (with base slug templates) that somehow included in every single post's head a link to /templates/post/feed/. I've removed this link to the feed from the head, but want to block robots ...
Steve's user avatar
  • 1,677
3 votes
2 answers
9k views

Why does do_robots() Allow: /wp-admin/admin-ajax.php by default?

This is happening with a fresh, new WordPress install. But all you need to do is look at the crazy do_robots() function, which outputs the following... User-agent: * Disallow: /wp-admin/ Allow: /wp-...
PJ Brunet's user avatar
  • 500
0 votes
0 answers
310 views

WordPress site using Yoast SEO being blocked from indexing by robots.txt

Morning all! I've just built a website for a client, using WordPress and the Yoast SEO plugin (free version). https://www.fly2help.org/ Whilst they're really happy with the website, I've noticed an ...
James's user avatar
  • 101
0 votes
0 answers
53 views

How to stop google from indexing spammed internal search results?

Google is indexing all kinds of spammed internal search results from my wordpress site, and I can't figure out how to put a stop to it. I added Disallow: /search/ to my robots.txt file, but that didn'...
Jon Fergus's user avatar
1 vote
1 answer
805 views

Set noindex page-comment from Pages 2, 3 and More?

I use All in one seo pack plugin. I see that Google indexed page comments (that are a pages generated from paginated comments). The example url are the followings: URL canonical - mydomain.com/url-...
Giulio's user avatar
  • 51
0 votes
4 answers
8k views

How to prevent Google from indexing the /wp-content/ directory?

I received an error from google saying that I have mobile usability errors on my site. the "page" that they are referring to is www.example.com/wp-content/plugins/ag-admin. I'm not asking ...
pressword's user avatar
  • 340
1 vote
1 answer
143 views

Why does Google Image Search still display my images?

I am having trouble with a few image files that I would like to block for Google Image search. I am using Yoast SEO Plugin, which allows me to directly edit the robots.txt file. A few people with ...
danschdev's user avatar
0 votes
0 answers
255 views

Bing/msn bots is heavily requesting random of my website

I am facing a big problem with my server. I have a website that keeps getting massive page requests coming from "Bing/Msn" bot every second or two and the ip changes now and then. Which is putting a ...
marouane91's user avatar
-1 votes
1 answer
2k views

How to noindex, follow a specific category wordpress?

Google appears to be ignoring my robots.txt (despite being valid) How do I noindex, follow a specific category without plugins?
PublicDisplayName's user avatar
0 votes
0 answers
569 views

Robots.txt file not updating

One of our website robots.txt file is not getting updated,deleted .htaccess and robots.txt file from root folder from one of the suggested answers Why does Google Webmaster Tools say my whole site is ...
Moin97's user avatar
  • 1
0 votes
1 answer
75 views

Why are some user agents still getting a meta robots tag with noindex from my site?

A friend pointed out to me that a website is not in Google because of the presence of a <meta name='robots' content='noindex,follow' /> tag in the home page. Ok, should be easy enough to turn ...
Pointy Haired Boss's user avatar
0 votes
1 answer
631 views

Where to put robot.txt and sitemap.xml for wordpress installed in its own directory

My website is setup where wordpress is installed in it's own subdirectory, but the content is served as though it's at the domain root. (This wordpress codex page details the configuration) Wordpress ...
SherylHohman's user avatar
0 votes
1 answer
50 views

Google saying is blocked by robots.txt without robots.txt on my website

Everything is in the title. I have no robots.txt on my website. I've enabled search engine indexing in the global settings. But still, Google cannot crawl my website due to an non-existing robots.txt. ...
Pierre's user avatar
  • 25
-1 votes
1 answer
23 views

How to view the code for a certain page?

Google is indexing pages that I used like a year ago and now they are 404. I want to add a noindex tag to them, but can't figure out how to open the code for a desired page so I can put in the tag. In ...
Ivailo Ivanov's user avatar
1 vote
1 answer
4k views

Disallow author with robots.txt

I would like to disallow the authors (of posts for example) in the file robots.txt. In fact, I noticed that for some of my sites, Google indexed the authors of articles, which I don't want to. Someone ...
ArbreMojo's user avatar
0 votes
1 answer
788 views

How to remove Wordpress Virtual Robots.txt that keeps overriding my root robots.txt? [closed]

So here is the issue. I am trying to remove the virtual robots.txt file in a wordpress site. I realized this was generated with a plugin (xml sitemap plugin) prior to me working on the site. Here is ...
Rob's user avatar
  • 44
1 vote
1 answer
53 views

install robots.txt to wordpress website admin

our website is http://syno-int.co.kr and using wordpress. I tried to install robots.txt to plugin section, but when I put zip file in it, it shows like this. Unable to create directory wp-content/...
Alicia's user avatar
  • 11
0 votes
1 answer
2k views

Old robots.txt file not changing, can't update to the current robots.txt

While my wordpress website was in production, I created a robots.txt file to disallow everything. When the site was ready, I deleted the robots.txt file through cpanel and never thought much about it. ...
Gan Zhe Jing's user avatar
1 vote
1 answer
1k views

Virtual robots.txt missing

on every WordPress installation there is a default virtual robots.txt file with the following User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php but on my 2 last project /robots.txt ...
David Horizonef's user avatar
-2 votes
2 answers
943 views

Google Webmaster Tools error "Url blocked by robots.txt."

I have a WordPress website, I submitted it to Google Webmaster and after submitting the sitemap, it's giving me error "URL blocked by robots.txt". Please help, what should I do?
Joel's user avatar
  • 1
3 votes
1 answer
4k views

Generating robots.txt dynamically

I have a subfolder WP installation. It serves multiple domains, which are linked to the same folder. The requested domain is received in wp-config.php (from the $_SERVER variable) and used to define ...
Igor Skoldin's user avatar
2 votes
3 answers
8k views

"No sitemap linked in your robots.txt file"

Relatively new to anything that doesn't require common sense in WordPress. My previous site was built poorly etc, so cutting a long story short I installed a new theme and started building up pages ...
Josh's user avatar
  • 21
0 votes
1 answer
97 views

Question about robots.txt file on WordPress site

I don't know why, but Google started indexing some pages that we don't have on our server. Mostly pages with the ?tag=22&page=30 query string. By the way, we don't have any "tag", and as result ...
David  D's user avatar
  • 11
3 votes
2 answers
308 views

Correct way to hide pseudo pages from being shown?

The homepage on a site we've developed has four pieces of content: the parent page and three further "pages" of text which we bring into the parent page using PHP. Call them Foo, Bar1, Bar2, Bar3 if ...
hawbsl's user avatar
  • 490
7 votes
4 answers
9k views

Robots.txt not updating

When I try to google search my web address, the follow comes up as the meta description "A description for this result is not available because of this site's robots.txt – learn more." The ...
Confused's user avatar
-1 votes
1 answer
38 views

No exposure on search engines

I have a blog but it doesn't show on any search engine. Why can't my wordpress blog be seen but I am always publishing.
Nnamdi Ebo's user avatar
1 vote
1 answer
3k views

robots.txt Not Updating

Today I received an email from Webmaster Tools stating that "Googlebot cannot access CSS and JS files". So I set about updating my robots.txt file located in the root of my server. Original Contents: ...
user1356029's user avatar
0 votes
2 answers
2k views

Disallow: /wp-* in robots.txt?

I have just received an email from Google advising that it can't accessing certain javascript and css content from my site. I've looked at the robots.txt file which contains: User-agent: * ...
ban-geoengineering's user avatar
1 vote
1 answer
68 views

Where is the "Privacy" menu for the "WP Robots Txt" plugin?

I want to be able to update the robots.txt for my WordPress site. From brief research it looks like the "WP Robots Txt" plugin is the easiest. However the tutorials eg https://www.youtube.com/watch?v=...
Chris's user avatar
  • 201
1 vote
0 answers
333 views

Robots.txt with WordPress and Joomla in same server

I have this configuration in my server root (hosted): root -> a Wordpress blog DirectoryJ -> a Joomla site DirectoryWP -> Another Wordpress blog I don't know how to handle the robots.txt. Joomla 3.4 ...
Jules's user avatar
  • 11
0 votes
1 answer
2k views

What is the meaning of User-agent and Disallow in robots.txt file in wordpress

What is the meaning of User-agent and Disallow in robots.txt file in WordPress. I am little bit confused in this.
TrendyUpdates's user avatar
0 votes
1 answer
371 views

Trying to update robots.txt [closed]

I have installed and activaded the WP Robots txt plugin. Then I have gone to wp-admin/options-reading.php I have pasted a new robotst.txt content and Save it. But I can not see the new content at ...
tirenweb's user avatar
  • 101
6 votes
2 answers
2k views

NoFollow Entire Website

By default whenever you disable indexing via Admin Settings [ x ] Discourage search engines from indexing this site It adds a meta tag in the header like so: <meta name='robots' content='...
Howdy_McGee's user avatar
  • 20.5k
4 votes
1 answer
1k views

Stuck on my server root folder, robots.txt file not deleting

I remove robots.txt file in my server root but google can't delete that. Webmastertool to create robotx.txt file How can I remove or disable.
Irfan's user avatar
  • 62
-1 votes
1 answer
2k views

Google is unable to crawl Robots.txt file of my website and please help me

Today I got a mail from Google webmasters, that Google bots are unable to find robots.txt file of my website. I also tried to locate the file by http://www.mydomain.com/robots.txt but the file isn't ...
Gurjyot Singh's user avatar
3 votes
2 answers
5k views

Unwanted "crawl delay: 10" line added to my robots.txt

I've noticed that when I make a request to my robots.txt page, I get it with a crawl delay: 10 line added to it. I double-checked the file, and it doesn't have that line, so why does it appear? Is it ...
LordSilver's user avatar
1 vote
3 answers
6k views

Why does Google Webmaster Tools say my whole site is blocked by robots.txt when I have no robots.txt?

I submitted a sitemap for my WordPress site to Google Webmaster Tools. Several times. I keep getting the same messages: "Url blocked by robots.txt." and "Sitemap contains urls which are blocked by ...
user avatar
0 votes
1 answer
1k views

Disable Ajax for Spiders

We have a site on wpegnine that gets about 20k visits a day and heavily depends on ajax for our inbound marketing strategy (lead/visitor processing). These ajax calls are killing our server ...
atwellpub's user avatar
  • 164
0 votes
1 answer
163 views

Hide pages from google and visitors

I have wordpress site where i have few custom type pages. I load the content of those pages with jquery load. But I dont want that those pages are directly visible. So if user goes to http://www....
wpneebie's user avatar
1 vote
1 answer
3k views

How to solve Robots.txt problem with WordPress fresh install?

I installed a fresh copy of WordPress few weeks ago and I have been having some strange problems. 1- I get this message : "Comments for FirstName LastName name-last.com/‎ A description for this ...
0bserver07's user avatar
0 votes
2 answers
3k views

Add robots.txt to root

I have a new Wordpress 3.5.1 install (hosted on Dreamhost FWIW) that I do not want to be indexed by search engines. I would like to serve a simple robots.txt with Disallow: / for all user agents. I ...
Mike Deck's user avatar
  • 117
2 votes
1 answer
5k views

How to stop WordPress showing a virtual Robots.txt?

I can see the robots.txt file here : http://persian.cc/robots.txt But Google can't ! I know that I can find robots.txt files in the root of a website, but hey there is no robots.txt file in mine ...
Parsa Pejvak's user avatar
0 votes
2 answers
309 views

Is this robots.txt making Google not to show some of my content?

http://chineselearnonline.com/ User-agent: * Disallow: /feed/ Disallow: /trackback/ Disallow: /wp-admin/ Disallow: /wp-content/ Disallow: /wp-includes/ Disallow: /xmlrpc.php Disallow: /wp- Sitemap: ...
wyc's user avatar
  • 3,837
0 votes
1 answer
147 views

wp_redirect and search engine bots

I have some pages and categories you have to be looged in to have access to. I am using wp_redirect to send anyone who is not logged in to the front page if they try to open those pages. Is it enough ...
luiquao's user avatar
  • 161
1 vote
1 answer
286 views

Worried I have a funky .htaccess for WP site in light of strange search engine behavior

My friend started a new (non-commercial) blog 5+ months ago, but Google + Bing are acting like we're telling it to not be indexed. I don't mean it ranks poorly. I mean it doesn't rank at all. I ...
jose's user avatar
  • 11