0

I don't know why, but Google started indexing some pages that we don't have on our server. Mostly pages with the ?tag=22&page=30 query string. By the way, we don't have any "tag", and as result number of 404 pages increased on the site's Google Search Console.

?tag=22&page=30
?tag=27&page=530
?tag=27&page=600
?tag=309&page=81

How can I close/disallow access to this files?

I've added this line, but I'm not sure if it will work:

Disallow: /?tag*
  • Please improve your question title. Summarize your specific problem, do not include the word WordPress. – fuxia Sep 21 '16 at 19:33
  • Downvoted and close voted as this is just a generic robot.txt question with nothing specific to wordpress in it. anyway don't look at your logs and you will live longer and happier. just because someone put some crap url somewhere do not mean you need to spend time "fixing" it. returning 404 is exactly what should be done and nothing more – Mark Kaplun Sep 22 '16 at 3:12
0

Here is some info about robots.txt. It looks like your example wouldn't work correctly with the * but, it looks to me like you'd be ok if you just removed the *.

You can always check the robots.txt at with a validator like this one

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.