Here's the JS I use to replace checkboxes for radio buttons.
function checktoradio(){
echo '<script type="text/javascript">jQuery("#categorychecklist-pop input, #categorychecklist input, .cat-checklist input").each(function(){this.type="radio"});</script>';
}
add_action('admin_footer', 'checktoradio');
This would also work for custom terms, you just need to find out the custom term ID used by inspecting the metabox element.
You would place this code inside functions.php. It basically adds a jquery script to the footer of the admin pages.
Disclaimer: I found the original code a long time ago on some website and adapted for my own use.