I use the category/postname permalink structure and my site contains over 1200 posts and around 100 pages.
It loads in around 1 second on average and i have thousands of images and Javascript for Github Gists on most posts.
There's 2 areas i think are amongst the most important for performance and efficiency.
In my opinion, the best solution to solve performance issues is to host on managed servers. Could never get my site to load in less than 3.5 seconds using different levels of VPS.
I have also found that theme framework selection can make a big difference. After moving many sites over to a more efficient parent theme framework from inefficient themes it was clear choosing a good parent theme framework can make a big difference in many cases.
Poor quality custom coding in your theme and plugins can also cause problems.
It doesn't matter how many posts your site includes, its how many are being loaded, what the page contains, like images, Javascript etc and how your servers handle this as far as how they are managed and configured.
Caching
Based on my experience using the most popular caching plugins for WordPress, they don't perform nearly as well as a Managed Proprietary Caching System which includes:
- Page Caching
- Object Caching - WordPress already includes a caching system which is better managed by proprietary caching than plugins.
- CDN management and URL re-writing - Caching plugins like W3 Total Cache rewrite only about 50% of your URLs compared to the best proprietary caching systems.
- Browser Headers and Compression