What is the meaning of User-agent and Disallow in robots.txt file in WordPress. I am little bit confused in this.
Add a comment
|
1 Answer
User-agent
They are Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses.
Disallow
It tells the robots that it should not visit the pages or directory of site which are not allowed by this tag.
