Simple answer is - always. Cached site is simply faster than non-cached, that is better on all fronts - performance, resource consumption, user experience.
More complex answer - you should always use properly configured caching that fits your needs, size of site, traffic and available hosting resources.
For examples:
- blanket static cache can do more harm than good in some situations, I think larger sites use more selective approach on what to cache (I vaguely remember reading wordpress.com does that with batcache);
- serving gzipped content is widely accepted practice, but it trades bandwidth for CPU and can cause processor load issues accordingly.
Really caching and performance is giant topic. I barely scratched it myself.
My personal advices:
- Read the docs. Everyone using caching plugins needs to have at least basic understanding of what is going on.
- Don't be greedy with advanced toys. Some options are simply put for big league, shared hosting or even VPS with slow disk can suffer from them.
- Ask your hosting. If they have any kind of decent support in place they will be able to come up with at least basic recommendation of what to use so it plays well with other stuff on server and doesn't cause overload.