Tag Info

New answers tagged

3

A lot of time since this quesiton and answer were posted. Since then things has changed a lot. The typical recommendation about disallow crawlers to access wp-content/themes, wp-content/plugins, wp-includes, etc are no longer valid. For example, lets talk about Google. Googlebot was rendering websites without CSS and without js, but not actually. Actually ...


1

You should follow Joost de Valk's current approach where very little is blocked in robots.txt, but also understand that each site will have a uniquely appropriate policy that will need to be reviewed and changed over time. Many of the answers given here previously are dated and will result in SEO self-sabotage since Google checks for "mobile friendliness" ...



Top 50 recent answers are included