Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [robots.txt]

This is a file that contains directions for search spiders. Its content depends on whether the user has allowed search engines to crawl the site.

-2
votes
2answers
859 views

Google Webmaster Tools error “Url blocked by robots.txt.”

I have a WordPress website, I submitted it to Google Webmaster and after submitting the sitemap, it's giving me error "URL blocked by robots.txt". Please help, what should I do?
1
vote
1answer
39 views

install robots.txt to wordpress website admin

our website is http://syno-int.co.kr and using wordpress. I tried to install robots.txt to plugin section, but when I put zip file in it, it shows like this. Unable to create directory wp-content/...
1
vote
1answer
485 views

Virtual robots.txt missing

on every WordPress installation there is a default virtual robots.txt file with the following User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php but on my 2 last project /robots.txt ...
0
votes
1answer
21 views

Why are some user agents still getting a meta robots tag with noindex from my site?

A friend pointed out to me that a website is not in Google because of the presence of a <meta name='robots' content='noindex,follow' /> tag in the home page. Ok, should be easy enough to turn ...
0
votes
1answer
24 views

Google saying is blocked by robots.txt without robots.txt on my website

Everything is in the title. I have no robots.txt on my website. I've enabled search engine indexing in the global settings. But still, Google cannot crawl my website due to an non-existing robots.txt. ...
0
votes
1answer
48 views

Question about robots.txt file on WordPress site

I don't know why, but Google started indexing some pages that we don't have on our server. Mostly pages with the ?tag=22&page=30 query string. By the way, we don't have any "tag", and as result ...
0
votes
1answer
1k views

Disable Ajax for Spiders

We have a site on wpegnine that gets about 20k visits a day and heavily depends on ajax for our inbound marketing strategy (lead/visitor processing). These ajax calls are killing our server ...
0
votes
1answer
125 views

Hide pages from google and visitors

I have wordpress site where i have few custom type pages. I load the content of those pages with jquery load. But I dont want that those pages are directly visible. So if user goes to http://www....
0
votes
1answer
136 views

wp_redirect and search engine bots

I have some pages and categories you have to be looged in to have access to. I am using wp_redirect to send anyone who is not logged in to the front page if they try to open those pages. Is it enough ...
-1
votes
1answer
22 views

How to view the code for a certain page?

Google is indexing pages that I used like a year ago and now they are 404. I want to add a noindex tag to them, but can't figure out how to open the code for a desired page so I can put in the tag. In ...
0
votes
0answers
45 views

Robots.txt file not updating

One of our website robots.txt file is not getting updated,deleted .htaccess and robots.txt file from root folder from one of the suggested answers Why does Google Webmaster Tools say my whole site is ...
0
votes
0answers
23 views

New Index coverage related issue for all the “Tag” pages in Google Webmaster tool

I have a blogging website in Wordpress. I am using Yoast SEO for sitemaps. I got the following error message in Google webmaster tool: Search Console has identified that your site is affected by 1 ...
0
votes
0answers
289 views

Robots.txt with WordPress and Joomla in same server

I have this configuration in my server root (hosted): root -> a Wordpress blog DirectoryJ -> a Joomla site DirectoryWP -> Another Wordpress blog I don't know how to handle the robots.txt. Joomla 3.4 ...