I have profiled my blog with the P3 Plugin and it pointed out that Yoast SEO slows down the site a bit. I then found out that the biggest performance penalty is paid when you "force rewrite" the titles, which I am doing.
Now I do have quite an aggressive caching turned on with WP Super Cache. Does it still matter if I force-rewrite the titles then? Or in my case it doesn't change a thing since pages are pre-generated and served as static content?