Questions tagged [robots.txt]
This is a file that contains directions for search spiders. Its content depends on whether the user has allowed search engines to crawl the site.
54
questions
0
votes
0
answers
13
views
How to dissallow all wordpress posts in robots.txt?
Well, I have a question. How can I dissallow posts in the robots.txt file?
My current robots.txt file:
User-agent: Scrapy
Allow: /
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
...
- 1
0
votes
0
answers
14
views
Hide plugins and theme from public
I like to hide all or some of my plugins and themes and CMS from public such that the users cannot detect which plugins/themes/CMS I used to build my website.
Please let me know if it is possible to ...
- 103
0
votes
1
answer
38
views
Use a filter for wp_robots to block CPT/feed/
We have a CPT (with base slug templates) that somehow included in every single post's head a link to /templates/post/feed/.
I've removed this link to the feed from the head, but want to block robots ...
- 1,588
0
votes
2
answers
1k
views
Why does do_robots() Allow: /wp-admin/admin-ajax.php by default?
This is happening with a fresh, new WordPress install. But all you need to do is look at the crazy do_robots() function, which outputs the following...
User-agent: *
Disallow: /wp-admin/
Allow: /wp-...
- 431
0
votes
0
answers
50
views
WordPress site using Yoast SEO being blocked from indexing by robots.txt
Morning all! I've just built a website for a client, using WordPress and the Yoast SEO plugin (free version).
https://www.fly2help.org/
Whilst they're really happy with the website, I've noticed an ...
- 101
0
votes
0
answers
35
views
How to stop google from indexing spammed internal search results?
Google is indexing all kinds of spammed internal search results from my wordpress site, and I can't figure out how to put a stop to it. I added Disallow: /search/ to my robots.txt file, but that didn'...
- 79
1
vote
1
answer
428
views
Set noindex page-comment from Pages 2, 3 and More?
I use All in one seo pack plugin.
I see that Google indexed page comments (that are a pages generated from paginated comments).
The example url are the followings:
URL canonical - mydomain.com/url-...
- 51
0
votes
3
answers
5k
views
How to prevent Google from indexing the /wp-content/ directory?
I received an error from google saying that I have mobile usability errors on my site.
the "page" that they are referring to is www.example.com/wp-content/plugins/ag-admin.
I'm not asking ...
- 340
1
vote
1
answer
115
views
Why does Google Image Search still display my images?
I am having trouble with a few image files that I would like to block for Google Image search.
I am using Yoast SEO Plugin, which allows me to directly edit the robots.txt file.
A few people with ...
- 11
0
votes
0
answers
141
views
Bing/msn bots is heavily requesting random of my website
I am facing a big problem with my server. I have a website that keeps getting massive page requests coming from "Bing/Msn" bot every second or two and the ip changes now and then. Which is putting a ...
- 17
0
votes
1
answer
1k
views
How to noindex, follow a specific category wordpress?
Google appears to be ignoring my robots.txt (despite being valid)
How do I noindex, follow a specific category without plugins?
0
votes
0
answers
520
views
Robots.txt file not updating
One of our website robots.txt file is not getting updated,deleted .htaccess and robots.txt file from root folder from one of the suggested answers
Why does Google Webmaster Tools say my whole site is ...
- 1
0
votes
1
answer
71
views
Why are some user agents still getting a meta robots tag with noindex from my site?
A friend pointed out to me that a website is not in Google because of the presence of a
<meta name='robots' content='noindex,follow' />
tag in the home page. Ok, should be easy enough to turn ...
0
votes
1
answer
506
views
Where to put robot.txt and sitemap.xml for wordpress installed in its own directory
My website is setup where wordpress is installed in it's own subdirectory, but the content is served as though it's at the domain root.
(This wordpress codex page details the configuration)
Wordpress ...
- 659
0
votes
1
answer
45
views
Google saying is blocked by robots.txt without robots.txt on my website
Everything is in the title.
I have no robots.txt on my website. I've enabled search engine indexing in the global settings. But still, Google cannot crawl my website due to an non-existing robots.txt.
...
- 25
-1
votes
1
answer
23
views
How to view the code for a certain page?
Google is indexing pages that I used like a year ago and now they are 404. I want to add a noindex tag to them, but can't figure out how to open the code for a desired page so I can put in the tag. In ...
1
vote
1
answer
3k
views
Disallow author with robots.txt
I would like to disallow the authors (of posts for example) in the file robots.txt. In fact, I noticed that for some of my sites, Google indexed the authors of articles, which I don't want to. Someone ...
- 21
0
votes
1
answer
703
views
How to remove Wordpress Virtual Robots.txt that keeps overriding my root robots.txt? [closed]
So here is the issue. I am trying to remove the virtual robots.txt file in a wordpress site. I realized this was generated with a plugin (xml sitemap plugin) prior to me working on the site.
Here is ...
- 44
1
vote
1
answer
44
views
install robots.txt to wordpress website admin
our website is http://syno-int.co.kr
and using wordpress.
I tried to install robots.txt to plugin section,
but when I put zip file in it, it shows like this.
Unable to create directory wp-content/...
- 11
0
votes
1
answer
2k
views
Old robots.txt file not changing, can't update to the current robots.txt
While my wordpress website was in production, I created a robots.txt file to disallow everything. When the site was ready, I deleted the robots.txt file through cpanel and never thought much about it. ...
1
vote
1
answer
1k
views
Virtual robots.txt missing
on every WordPress installation there is a default virtual robots.txt file with the following
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
but on my 2 last project /robots.txt ...
-2
votes
2
answers
941
views
Google Webmaster Tools error "Url blocked by robots.txt."
I have a WordPress website, I submitted it to Google Webmaster and after submitting the sitemap, it's giving me error "URL blocked by robots.txt".
Please help, what should I do?
- 1
3
votes
1
answer
3k
views
Generating robots.txt dynamically
I have a subfolder WP installation. It serves multiple domains, which are linked to the same folder. The requested domain is received in wp-config.php (from the $_SERVER variable) and used to define ...
- 391
2
votes
3
answers
7k
views
"No sitemap linked in your robots.txt file"
Relatively new to anything that doesn't require common sense in WordPress. My previous site was built poorly etc, so cutting a long story short I installed a new theme and started building up pages ...
- 21
0
votes
1
answer
80
views
Question about robots.txt file on WordPress site
I don't know why, but Google started indexing some pages that we don't have on our server. Mostly pages with the ?tag=22&page=30 query string. By the way, we don't have any "tag", and as result ...
- 11
3
votes
2
answers
306
views
Correct way to hide pseudo pages from being shown?
The homepage on a site we've developed has four pieces of content: the parent page and three further "pages" of text which we bring into the parent page using PHP. Call them Foo, Bar1, Bar2, Bar3 if ...
- 460
7
votes
4
answers
8k
views
Robots.txt not updating
When I try to google search my web address, the follow comes up as the meta description
"A description for this result is not available because of this site's
robots.txt – learn more."
The ...
- 81
-1
votes
1
answer
38
views
No exposure on search engines
I have a blog but it doesn't show on any search engine.
Why can't my wordpress blog be seen but I am always publishing.
1
vote
1
answer
3k
views
robots.txt Not Updating
Today I received an email from Webmaster Tools stating that "Googlebot cannot access CSS and JS files". So I set about updating my robots.txt file located in the root of my server.
Original Contents:
...
- 111
0
votes
2
answers
2k
views
Disallow: /wp-* in robots.txt?
I have just received an email from Google advising that it can't accessing certain javascript and css content from my site.
I've looked at the robots.txt file which contains:
User-agent: *
...
1
vote
1
answer
68
views
Where is the "Privacy" menu for the "WP Robots Txt" plugin?
I want to be able to update the robots.txt for my WordPress site.
From brief research it looks like the "WP Robots Txt" plugin is the easiest. However the tutorials eg https://www.youtube.com/watch?v=...
- 181
1
vote
0
answers
332
views
Robots.txt with WordPress and Joomla in same server
I have this configuration in my server root (hosted):
root -> a Wordpress blog
DirectoryJ -> a Joomla site
DirectoryWP -> Another Wordpress blog
I don't know how to handle the robots.txt. Joomla 3.4 ...
- 11
0
votes
1
answer
2k
views
What is the meaning of User-agent and Disallow in robots.txt file in wordpress
What is the meaning of User-agent and Disallow in robots.txt file in WordPress. I am little bit confused in this.
0
votes
1
answer
328
views
Trying to update robots.txt [closed]
I have installed and activaded the WP Robots txt plugin. Then I have gone to wp-admin/options-reading.php I have pasted a new robotst.txt content and Save it. But I can not see the new content at ...
- 101
6
votes
2
answers
1k
views
NoFollow Entire Website
By default whenever you disable indexing via Admin Settings
[ x ] Discourage search engines from indexing this site
It adds a meta tag in the header like so:
<meta name='robots' content='...
- 19.8k
4
votes
1
answer
959
views
Stuck on my server root folder, robots.txt file not deleting
I remove robots.txt file in my server root but google can't delete that.
Webmastertool to create robotx.txt file
How can I remove or disable.
- 62
-1
votes
1
answer
2k
views
Google is unable to crawl Robots.txt file of my website and please help me
Today I got a mail from Google webmasters, that Google bots are unable to find robots.txt file of my website. I also tried to locate the file by http://www.mydomain.com/robots.txt but the file isn't ...
3
votes
2
answers
5k
views
Unwanted "crawl delay: 10" line added to my robots.txt
I've noticed that when I make a request to my robots.txt page, I get it with a crawl delay: 10 line added to it. I double-checked the file, and it doesn't have that line, so why does it appear? Is it ...
- 49
1
vote
3
answers
6k
views
Why does Google Webmaster Tools say my whole site is blocked by robots.txt when I have no robots.txt?
I submitted a sitemap for my WordPress site to Google Webmaster Tools. Several times. I keep getting the same messages:
"Url blocked by robots.txt." and "Sitemap contains urls which are blocked by ...
- 11
0
votes
1
answer
1k
views
Disable Ajax for Spiders
We have a site on wpegnine that gets about 20k visits a day and heavily depends on ajax for our inbound marketing strategy (lead/visitor processing).
These ajax calls are killing our server ...
- 154
0
votes
1
answer
159
views
Hide pages from google and visitors
I have wordpress site where i have few custom type pages. I load the content of those pages with jquery load. But I dont want that those pages are directly visible. So if user goes to http://www....
- 1
1
vote
1
answer
3k
views
How to solve Robots.txt problem with WordPress fresh install?
I installed a fresh copy of WordPress few weeks ago and I have been having some strange problems.
1- I get this message : "Comments for FirstName LastName
name-last.com/
A description for this ...
- 113
0
votes
2
answers
3k
views
Add robots.txt to root
I have a new Wordpress 3.5.1 install (hosted on Dreamhost FWIW) that I do not want to be indexed by search engines. I would like to serve a simple robots.txt with Disallow: / for all user agents.
I ...
- 117
2
votes
1
answer
4k
views
How to stop WordPress showing a virtual Robots.txt?
I can see the robots.txt file here :
http://persian.cc/robots.txt
But Google can't !
I know that I can find robots.txt files in the root of a website, but hey there is no robots.txt file in mine ...
- 31
0
votes
2
answers
308
views
Is this robots.txt making Google not to show some of my content?
http://chineselearnonline.com/
User-agent: *
Disallow: /feed/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /xmlrpc.php
Disallow: /wp-
Sitemap: ...
- 3,717
0
votes
1
answer
145
views
wp_redirect and search engine bots
I have some pages and categories you have to be looged in to have access to. I am using wp_redirect to send anyone who is not logged in to the front page if they try to open those pages. Is it enough ...
- 161
1
vote
1
answer
284
views
Worried I have a funky .htaccess for WP site in light of strange search engine behavior
My friend started a new (non-commercial) blog 5+ months ago, but Google + Bing are acting like we're telling it to not be indexed.
I don't mean it ranks poorly. I mean it doesn't rank at all.
I ...
- 11
3
votes
2
answers
3k
views
Create unique robots.txt for every site on multisite-installation
Does anyone know how I can have a unique robots.txt file for every domain served by my wp-multisite-installation? I did a search on plugins but couldn't find anything suitable.
- 33
0
votes
2
answers
733
views
Can you please verify my robots.txt for wordpress
I am a newb in writing this and since this is dangerous stuff to fidlle with (if you are not sure what you are doing) can you please verify it?
I use wordpress multisite, this is why I chose to allow ...
- 311
0
votes
1
answer
1k
views
auto generated robots.txt problem
I have a site visibility in privacy settings set to allow search engines to see my site but for some reason when i go to the auto generated robots.txt file it shows
User-agent: *
Disallow:
switching ...
- 2,984