WordPress 4.3 and below
There's nothing in the template hierarchy for showing a template based on a post ID unless you're using the page
post type.
Instead you're going to have to place conditional logic at the top of your single-portfolio.php
.
I would also recommend you use meta options, terms, or the post slug instead of the ID.If you use post IDs, a simple import and export of the content would break your theme.
Also consider implementing a metabox with a dropdown that allows you to choose a template. This way you're not restricting yourself and hardcoding content in your theme, and you can build reusable general templates for certain portfolio items
WordPress 4.4
4.4 adds a new template to the hierarchy that takes the form:
single-{post_type}-{post_name}.php
A new theme template has been added to the theme hierarchy as of r34800: single-{post_type}-{post_name}.php. This template follows the rules of is_single() and is used for a single post or custom post type. It’s most useful for targeting a specific post in a custom post type, and brings consistency to the templates available for pages and taxonomies. It comes in the hierarchy before single.php and single-{post_type}.php.
Don’t forget, too, that in WordPress 4.3 singular.php was introduced and the hierarchy of attachment templates was fixed.