0

While my wordpress website was in production, I created a robots.txt file to disallow everything. When the site was ready, I deleted the robots.txt file through cpanel and never thought much about it. Recently, I realized that the website was not showing up on google search results, upon further investigation, i realized that the old robots.txt file was still there (even though I can't locate the file in my root folder).

http://youngauthors.my/robots.txt

I inserted another robots.txt file in my root folder, this time with the allow option instead of disallow. But it doesn't update and still blocks google from accessing the site.

Current robots.txt in root folder

Current robots.txt content

Old robots.txt as you type the url (this file was deleted a long time ago)

Even in the yoast editor it's the current file, but the url still brings up the old file

1

Just delete file robots.txt. WordPress is generating virtual robots.txt file, and Yoast SEO plugin (and other plugins too) allows you to customize it. But, if you have an actual robots.txt file, it will prevent virtual one to work.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.