What is the meaning of User-agent and Disallow in robots.txt file in WordPress. I am little bit confused in this.

1 Answer 1



They are Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses.


It tells the robots that it should not visit the pages or directory of site which are not allowed by this tag.

Details here

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.