Pretty much as the title says, I want to use the "link" button from the HTML mode in the Visual Mode in the WYSIWYG editor (the link/unlink in visual is to "advanced" for my liking). Is this possible without too much of a hassle?
Sorry, but no. :) You can't simply use the html-button in the wysiwyg-editor. Both are different programms and have nothing in common except for their location. You'd have to unregister the current buttons, write code for a new one (that fits your likings) and then add this one. You'll find hundreds of tutorials for adding buttons to tinyMCE both on google, as here in WPSE.