Before I start, it will be a good idea to check out my answer to the following post
It will give you some more insight
As I stated in the linked answer, You can have a fully functional website with just
index.php as template to display posts with for all pages
What you are doing is fine, and if there are any speed difference, it should be very very minute. I would however stress that your approach might have the drawback of one templates having to do to much.
Because you would want to do things differently for different pages, it will be a good idea to create those templates needed the relevant page loads. As I said, if there is any speed difference with any method, it would be really very very minute which you will not worry about. Having different templates for different pages with different layouts do have the advantage of that you do not overload one template with a lot of unnecessary code.
For extra reference, see my answer to the following post as well