Can I add some custom fields to specific taxonomy so when ever I call that taxonomy, custom fields attached to that taxonomy should also appear.
You can hook into the edit form field action of that specific taxonomy.
If you look in /wp-admin/edit-tag-form.php there are hooks for every taxonomy. Starting at Line 69:
So for a custom taxonomy, it would be $taxonomy_edit_form_fields with $taxonomy being the taxonomies name. So you would add the hook, within the function, add your form fields. Then you also need to hook into the 'edit_term' action and if your form fields aren't empty then save it as an option using update_option().
This won't allow you to just "call the taxonomy" and have them already attached to the object, but you can loop through your terms once you get them and get the option associated with that term very easily.