You should use the svn tags to control which version of your plugin is made available through the plugins directory. Make sure there is a Stable tag:
header in your plugin's readme.txt
and set it to which ever tag you want to use. This page describes how to use the WordPress svn server to control your plugin.
However, I am not sure that there is a way to remove a plugin for the directory. Perhaps you could revert to a working revision, or at least a revision that is not causing damage to existing installs, and make that the stable version. If you have just started development you could always just remove everything but the bare minimum from you plugin, commit it to svn, tag as 0.1 or something and set stable tag to 0.1 and the commit everything. This should suffice to get you started:
plugin/$: vi trunk/plugin.php # make sure plugin is in basic working order
plugin/$: vi trunk/readme.txt # set stable tag to 0.1
plugin/$: svn ci -m "Most basic version of plugin"
plugin/$: svn cp trunk tags/0.1
plugin/$: svn ci -m "Tagging version 0.1"