If it's getting too hard and complicated, you can simply add new buttons by using jQuery. Simply clone an existing or create a new button, and append it to the editor's toolbar. You can wrap the javascript with a php function, and run it in the admin footer or something.
Or you can use the edButton function. Here is a dirty and fast written example for adding p and pre buttons.
// Add buttons to html editor
add_action('admin_print_footer_scripts','eg_quicktags');
function eg_quicktags() {
?>
<script type="text/javascript" charset="utf-8">
buttonA = edButtons.length;
edButtons[edButtons.length] = new edButton('ed_paragraph','p','<p>','</p><br />','p');
buttonB = edButtons.length;
edButtons[edButtons.length] = new edButton('ed_pre','pre','<pre lang="php">','</pre>','r');
jQuery(document).ready(function($){
jQuery("#ed_toolbar").append('<input type="button" value="p" id="ed_paragraph" class="ed_button" onclick="edInsertTag(edCanvas, buttonA);" title="p" />');
jQuery("#ed_toolbar").append('<input type="button" value="pre" id="ed_pre" class="ed_button" onclick="edInsertTag(edCanvas, buttonB);" title="pre" />');
});
</script>
<?php
}
EDIT: In Wordpress 3.3 (and above), the quicktag addition is changed. However, the edButton lazy solution is somehow working, some plugins might cancel it out.
The new and the right way of adding new buttons to html editor is like this :
// Add buttons to html editor
add_action('admin_print_footer_scripts','eg_quicktags');
function eg_quicktags() {
?>
<script type="text/javascript" charset="utf-8">
/* Adding Quicktag buttons to the editor Wordpress ver. 3.3 and above
* - Button HTML ID (required)
* - Button display, value="" attribute (required)
* - Opening Tag (required)
* - Closing Tag (required)
* - Access key, accesskey="" attribute for the button (optional)
* - Title, title="" attribute (optional)
* - Priority/position on bar, 1-9 = first, 11-19 = second, 21-29 = third, etc. (optional)
*/
QTags.addButton( 'eg_paragraph', 'p', '<p>', '</p>', 'p' );
QTags.addButton( 'eg_pre', 'pre','<pre lang="php">', '</pre>', 'q' );
</script>
<?php
}
I don't know if the QTags is added to the Wordpress Codex yet, so I added the required parameters in the comment section of the code.