Questions tagged [robots.txt]
This is a file that contains directions for search spiders. Its content depends on whether the user has allowed search engines to crawl the site.
48
questions
1
vote
1answer
49 views
Set noindex page-comment from Pages 2, 3 and More?
I use All in one seo pack plugin.
I see that Google indexed page comments (that are a pages generated from paginated comments).
The example url are the followings:
URL canonical - mydomain.com/url-...
0
votes
2answers
2k views
How to prevent Google from indexing the /wp-content/ directory?
I received an error from google saying that I have mobile usability errors on my site.
the "page" that they are referring to is www.example.com/wp-content/plugins/ag-admin.
I'm not asking ...
1
vote
1answer
68 views
Why does Google Image Search still display my images?
I am having trouble with a few image files that I would like to block for Google Image search.
I am using Yoast SEO Plugin, which allows me to directly edit the robots.txt file.
A few people with ...
0
votes
0answers
60 views
Bing/msn bots is heavily requesting random of my website
I am facing a big problem with my server. I have a website that keeps getting massive page requests coming from "Bing/Msn" bot every second or two and the ip changes now and then. Which is putting a ...
0
votes
1answer
637 views
How to noindex, follow a specific category wordpress?
Google appears to be ignoring my robots.txt (despite being valid)
How do I noindex, follow a specific category without plugins?
0
votes
0answers
387 views
Robots.txt file not updating
One of our website robots.txt file is not getting updated,deleted .htaccess and robots.txt file from root folder from one of the suggested answers
Why does Google Webmaster Tools say my whole site is ...
0
votes
1answer
52 views
Why are some user agents still getting a meta robots tag with noindex from my site?
A friend pointed out to me that a website is not in Google because of the presence of a
<meta name='robots' content='noindex,follow' />
tag in the home page. Ok, should be easy enough to turn ...
0
votes
1answer
377 views
Where to put robot.txt and sitemap.xml for wordpress installed in its own directory
My website is setup where wordpress is installed in it's own subdirectory, but the content is served as though it's at the domain root.
(This wordpress codex page details the configuration)
Wordpress ...
0
votes
1answer
38 views
Google saying is blocked by robots.txt without robots.txt on my website
Everything is in the title.
I have no robots.txt on my website. I've enabled search engine indexing in the global settings. But still, Google cannot crawl my website due to an non-existing robots.txt.
...
-1
votes
1answer
22 views
How to view the code for a certain page?
Google is indexing pages that I used like a year ago and now they are 404. I want to add a noindex tag to them, but can't figure out how to open the code for a desired page so I can put in the tag. In ...
0
votes
1answer
2k views
Disallow author with robots.txt
I would like to disallow the authors (of posts for example) in the file robots.txt. In fact, I noticed that for some of my sites, Google indexed the authors of articles, which I don't want to. Someone ...
0
votes
1answer
573 views
How to remove Wordpress Virtual Robots.txt that keeps overriding my root robots.txt? [closed]
So here is the issue. I am trying to remove the virtual robots.txt file in a wordpress site. I realized this was generated with a plugin (xml sitemap plugin) prior to me working on the site.
Here is ...
1
vote
1answer
41 views
install robots.txt to wordpress website admin
our website is http://syno-int.co.kr
and using wordpress.
I tried to install robots.txt to plugin section,
but when I put zip file in it, it shows like this.
Unable to create directory wp-content/...
0
votes
1answer
2k views
Old robots.txt file not changing, can't update to the current robots.txt
While my wordpress website was in production, I created a robots.txt file to disallow everything. When the site was ready, I deleted the robots.txt file through cpanel and never thought much about it. ...
1
vote
1answer
837 views
Virtual robots.txt missing
on every WordPress installation there is a default virtual robots.txt file with the following
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
but on my 2 last project /robots.txt ...
-2
votes
2answers
940 views
Google Webmaster Tools error “Url blocked by robots.txt.”
I have a WordPress website, I submitted it to Google Webmaster and after submitting the sitemap, it's giving me error "URL blocked by robots.txt".
Please help, what should I do?
3
votes
1answer
3k views
Generating robots.txt dynamically
I have a subfolder WP installation. It serves multiple domains, which are linked to the same folder. The requested domain is received in wp-config.php (from the $_SERVER variable) and used to define ...
2
votes
3answers
6k views
“No sitemap linked in your robots.txt file”
Relatively new to anything that doesn't require common sense in WordPress. My previous site was built poorly etc, so cutting a long story short I installed a new theme and started building up pages ...
0
votes
1answer
57 views
Question about robots.txt file on WordPress site
I don't know why, but Google started indexing some pages that we don't have on our server. Mostly pages with the ?tag=22&page=30 query string. By the way, we don't have any "tag", and as result ...
3
votes
2answers
299 views
Correct way to hide pseudo pages from being shown?
The homepage on a site we've developed has four pieces of content: the parent page and three further "pages" of text which we bring into the parent page using PHP. Call them Foo, Bar1, Bar2, Bar3 if ...
7
votes
4answers
7k views
Robots.txt not updating
When I try to google search my web address, the follow comes up as the meta description
"A description for this result is not available because of this site's
robots.txt – learn more."
The ...
-1
votes
1answer
37 views
No exposure on search engines
I have a blog but it doesn't show on any search engine.
Why can't my wordpress blog be seen but I am always publishing.
1
vote
1answer
2k views
robots.txt Not Updating
Today I received an email from Webmaster Tools stating that "Googlebot cannot access CSS and JS files". So I set about updating my robots.txt file located in the root of my server.
Original Contents:
...
0
votes
2answers
1k views
Disallow: /wp-* in robots.txt?
I have just received an email from Google advising that it can't accessing certain javascript and css content from my site.
I've looked at the robots.txt file which contains:
User-agent: *
...
1
vote
1answer
66 views
Where is the “Privacy” menu for the “WP Robots Txt” plugin?
I want to be able to update the robots.txt for my WordPress site.
From brief research it looks like the "WP Robots Txt" plugin is the easiest. However the tutorials eg https://www.youtube.com/watch?v=...
0
votes
0answers
328 views
Robots.txt with WordPress and Joomla in same server
I have this configuration in my server root (hosted):
root -> a Wordpress blog
DirectoryJ -> a Joomla site
DirectoryWP -> Another Wordpress blog
I don't know how to handle the robots.txt. Joomla 3.4 ...
0
votes
1answer
2k views
What is the meaning of User-agent and Disallow in robots.txt file in wordpress
What is the meaning of User-agent and Disallow in robots.txt file in WordPress. I am little bit confused in this.
0
votes
1answer
285 views
Trying to update robots.txt [closed]
I have installed and activaded the WP Robots txt plugin. Then I have gone to wp-admin/options-reading.php I have pasted a new robotst.txt content and Save it. But I can not see the new content at ...
6
votes
2answers
1k views
NoFollow Entire Website
By default whenever you disable indexing via Admin Settings
[ x ] Discourage search engines from indexing this site
It adds a meta tag in the header like so:
<meta name='robots' content='...
4
votes
1answer
837 views
Stuck on my server root folder, robots.txt file not deleting
I remove robots.txt file in my server root but google can't delete that.
Webmastertool to create robotx.txt file
How can I remove or disable.
-1
votes
1answer
2k views
Google is unable to crawl Robots.txt file of my website and please help me
Today I got a mail from Google webmasters, that Google bots are unable to find robots.txt file of my website. I also tried to locate the file by http://www.mydomain.com/robots.txt but the file isn't ...
3
votes
2answers
5k views
Unwanted “crawl delay: 10” line added to my robots.txt
I've noticed that when I make a request to my robots.txt page, I get it with a crawl delay: 10 line added to it. I double-checked the file, and it doesn't have that line, so why does it appear? Is it ...
1
vote
3answers
6k views
Why does Google Webmaster Tools say my whole site is blocked by robots.txt when I have no robots.txt?
I submitted a sitemap for my WordPress site to Google Webmaster Tools. Several times. I keep getting the same messages:
"Url blocked by robots.txt." and "Sitemap contains urls which are blocked by ...
0
votes
1answer
1k views
Disable Ajax for Spiders
We have a site on wpegnine that gets about 20k visits a day and heavily depends on ajax for our inbound marketing strategy (lead/visitor processing).
These ajax calls are killing our server ...
0
votes
1answer
147 views
Hide pages from google and visitors
I have wordpress site where i have few custom type pages. I load the content of those pages with jquery load. But I dont want that those pages are directly visible. So if user goes to http://www....
1
vote
1answer
2k views
How to solve Robots.txt problem with WordPress fresh install?
I installed a fresh copy of WordPress few weeks ago and I have been having some strange problems.
1- I get this message : "Comments for FirstName LastName
name-last.com/
A description for this ...
0
votes
2answers
2k views
Add robots.txt to root
I have a new Wordpress 3.5.1 install (hosted on Dreamhost FWIW) that I do not want to be indexed by search engines. I would like to serve a simple robots.txt with Disallow: / for all user agents.
I ...
1
vote
1answer
4k views
How to stop WordPress showing a virtual Robots.txt?
I can see the robots.txt file here :
http://persian.cc/robots.txt
But Google can't !
I know that I can find robots.txt files in the root of a website, but hey there is no robots.txt file in mine ...
0
votes
2answers
307 views
Is this robots.txt making Google not to show some of my content?
http://chineselearnonline.com/
User-agent: *
Disallow: /feed/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /xmlrpc.php
Disallow: /wp-
Sitemap: ...
0
votes
1answer
142 views
wp_redirect and search engine bots
I have some pages and categories you have to be looged in to have access to. I am using wp_redirect to send anyone who is not logged in to the front page if they try to open those pages. Is it enough ...
1
vote
1answer
274 views
Worried I have a funky .htaccess for WP site in light of strange search engine behavior
My friend started a new (non-commercial) blog 5+ months ago, but Google + Bing are acting like we're telling it to not be indexed.
I don't mean it ranks poorly. I mean it doesn't rank at all.
I ...
3
votes
2answers
3k views
Create unique robots.txt for every site on multisite-installation
Does anyone know how I can have a unique robots.txt file for every domain served by my wp-multisite-installation? I did a search on plugins but couldn't find anything suitable.
0
votes
2answers
709 views
Can you please verify my robots.txt for wordpress
I am a newb in writing this and since this is dangerous stuff to fidlle with (if you are not sure what you are doing) can you please verify it?
I use wordpress multisite, this is why I chose to allow ...
0
votes
1answer
1k views
auto generated robots.txt problem
I have a site visibility in privacy settings set to allow search engines to see my site but for some reason when i go to the auto generated robots.txt file it shows
User-agent: *
Disallow:
switching ...
1
vote
1answer
674 views
Questions about Robots.txt
I need some help with robots.txt.
Under Privacy Settings, I selected "I would like my blog to be visible to everyone, including search engines..."
As I understand it, WordPress automatically ...
0
votes
3answers
5k views
Setting up robots.txt with Yoast's Seo plugin
I was looking for a decent robots.txt solution for my wordpress sites and after some searching I found Yoast's SEO plugin that provides a complete SEO solution including indexation.
I set it up it to ...
7
votes
6answers
5k views
What is a good robots.txt?
What is the "best" setup for robots.txt?
I'm using the following permalink structure /%category%/%postname%/.
My robots.txt currently looks like this (copied from somewhere a long time ago):
User-...
1
vote
4answers
5k views
Preventing Search Engines Indexing Pages 2, 3 and More?
Do you know how to prevent indexing of pages past the home page in WP?
I mean I don't want mysite.com/page/2, mysite.com/page/3 to be indexed.
This is because I use home.php for my theme, so that ...
