0

What is the meaning of User-agent and Disallow in robots.txt file in WordPress. I am little bit confused in this.

1 Answer 1

1

User-agent

They are Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses.

Disallow

It tells the robots that it should not visit the pages or directory of site which are not allowed by this tag.

Details here

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.