Questions tagged [robots.txt]
This is a file that contains directions for search spiders. Its content depends on whether the user has allowed search engines to crawl the site.
14
questions with no upvoted or accepted answers
1
vote
1answer
41 views
install robots.txt to wordpress website admin
our website is http://syno-int.co.kr
and using wordpress.
I tried to install robots.txt to plugin section,
but when I put zip file in it, it shows like this.
Unable to create directory wp-content/...
1
vote
1answer
774 views
Virtual robots.txt missing
on every WordPress installation there is a default virtual robots.txt file with the following
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
but on my 2 last project /robots.txt ...
0
votes
0answers
53 views
Bing/msn bots is heavily requesting random of my website
I am facing a big problem with my server. I have a website that keeps getting massive page requests coming from "Bing/Msn" bot every second or two and the ip changes now and then. Which is putting a ...
0
votes
1answer
528 views
How to noindex, follow a specific category wordpress?
Google appears to be ignoring my robots.txt (despite being valid)
How do I noindex, follow a specific category without plugins?
0
votes
0answers
361 views
Robots.txt file not updating
One of our website robots.txt file is not getting updated,deleted .htaccess and robots.txt file from root folder from one of the suggested answers
Why does Google Webmaster Tools say my whole site is ...
0
votes
1answer
43 views
Why are some user agents still getting a meta robots tag with noindex from my site?
A friend pointed out to me that a website is not in Google because of the presence of a
<meta name='robots' content='noindex,follow' />
tag in the home page. Ok, should be easy enough to turn ...
0
votes
1answer
37 views
Google saying is blocked by robots.txt without robots.txt on my website
Everything is in the title.
I have no robots.txt on my website. I've enabled search engine indexing in the global settings. But still, Google cannot crawl my website due to an non-existing robots.txt.
...
0
votes
1answer
54 views
Question about robots.txt file on WordPress site
I don't know why, but Google started indexing some pages that we don't have on our server. Mostly pages with the ?tag=22&page=30 query string. By the way, we don't have any "tag", and as result ...
0
votes
0answers
300 views
Robots.txt with WordPress and Joomla in same server
I have this configuration in my server root (hosted):
root -> a Wordpress blog
DirectoryJ -> a Joomla site
DirectoryWP -> Another Wordpress blog
I don't know how to handle the robots.txt. Joomla 3.4 ...
0
votes
1answer
144 views
Hide pages from google and visitors
I have wordpress site where i have few custom type pages. I load the content of those pages with jquery load. But I dont want that those pages are directly visible. So if user goes to http://www....
0
votes
1answer
142 views
wp_redirect and search engine bots
I have some pages and categories you have to be looged in to have access to. I am using wp_redirect to send anyone who is not logged in to the front page if they try to open those pages. Is it enough ...
0
votes
1answer
1k views
Disable Ajax for Spiders
We have a site on wpegnine that gets about 20k visits a day and heavily depends on ajax for our inbound marketing strategy (lead/visitor processing).
These ajax calls are killing our server ...
-1
votes
1answer
22 views
How to view the code for a certain page?
Google is indexing pages that I used like a year ago and now they are 404. I want to add a noindex tag to them, but can't figure out how to open the code for a desired page so I can put in the tag. In ...
-2
votes
2answers
938 views
Google Webmaster Tools error “Url blocked by robots.txt.”
I have a WordPress website, I submitted it to Google Webmaster and after submitting the sitemap, it's giving me error "URL blocked by robots.txt".
Please help, what should I do?
