Tag Info

New answers tagged


A lot of time since this quesiton and answer were posted. Since then things has changed a lot. The typical recommendation about disallow crawlers to access wp-content/themes, wp-content/plugins, wp-includes, etc are no longer valid. For example, lets talk about Google. Googlebot was rendering websites without CSS and without js, but not actually. Actually ...


You should follow Joost de Valk's current approach where very little is blocked in robots.txt, but also understand that each site will have a uniquely appropriate policy that will need to be reviewed and changed over time. Many of the answers given here previously are dated and will result in SEO self-sabotage since Google checks for "mobile friendliness" ...

Top 50 recent answers are included