You need to use a few SQL queries, you can run them in something like PHPMyADMIN
DELETE FROM wp_posts WHERE `post_type` = 'customposttypename'
DELETE FROM wp_post_meta WHERE `meta_key` = 'metakeyname'
The above two will get you started. You may need to run other queries to clean up other entries but only you will know what other stuff you've inserted into your DB.
A word of warning! MAKE SURE YOU BACKUP YOUR DATABASE FIRST. And also before you run the DELETE query use a SELECT * FROM first to check whats returned and thats what you want to be deleted.