You don't indicate what type of hosting environment you're on (shared, VPS, co-location). Obviously this can have a major impact on your overall performance; for example a shared hosting environment typically is oversold by the vendor and resources are restricted.
Schedule a little down time for some benchmarking in your production environment or setup a duplicate test environment on the same physical host.
Factors that are important here -
- MySQL/database benchmarking
- Apache/httpd benchmarking
- OS benchmarking
- Network latency
Host if you have shell access:
- FirePHP with net tools and perhaps ySlow
So setup some simple monitoring/tools (i.e. top, mytop) to see how your host is performing:
- With the default Twenty Ten theme and no plugins enabled.
- Switch to your regular/custom theme.
- Benchmark each plugin's impact by enabling one plugin at a time. I'd suggest working from most important to least important.
This systematic approach should give you a pretty good idea of where your performance issues are. You may not be able to address issues while testing but you should be able to develop a plan of attack.