What's wrong in this query ? the sortby=name doesn't change anything.
$the_query = 'posts_per_page=-1&sortby=name&order=ASC&post_type=mycustom&custom_cat=mycat'; // query is made query_posts($the_query);
The "sort" query param is usually "orderby" not "sortby", to match the SQL
ORDER BY clause.
$the_query = 'posts_per_page=-1&orderby=name&order=ASC&post_type=mycustom&custom_cat=mycat';
It's also possible that the value you are passing is incorrect, but without knowing the details of your custom post type there's not much else I can suggest.