Cross-site Scripting Issues
You cannot transfer WP auth cookies between domains. You also don't want to store plaintext passwords for logging into another WP installation programmatically. So, you'll have to have users log into WordPress, and then access their login status via an API endpoint from the third-party site. This lets WordPress handle all the authentication. It is pretty secure as a user will have to physically login to the WP side in order for the API endpoint to serve up the data to the third-party.
Create an API Endpoint
Check out this article I just wrote here: http://coderrr.com/create-an-api-endpoint-in-wordpress/
Also, you can see the code demonstration here: https://gist.github.com/2982319
You'll have to figure out the logic for your own app needs, but this will allow you to create an endpoint where you can serve up anything you want from the WordPress side.
Since you're using WordPress as the authentication site, you can use a check like is_user_logged_in(). If they are logged in, return a user object to the third party with whatever information they need.
Logging in From The Third-Party
From the third-party, they can link to your login page for a seamless experience using the redirect_to query var. Once logged in, it will pass them back to the third-party site.
http://sub.yourdomain.com/wp-login.php?redirect_to=http%3A%2F%2Fwww.third-party-domain.com
Remote Logins
If you need to login users to WordPress from a third-party site, you can use some simple WP functions listed on this site: http://kuttler.eu/code/log-in-a-wordpress-user-programmatically/
You'll definitely need to use a shared secret and create time based hashes off that secret to keep things secure. Basically, here's what it would look like:
Third party sends request with a timestamp and a token generated by a shared secret:
$shared_secret = 'foobar'; //do not send this to the API endpoint
$timestamp = time();
$token = md5($shared_secret.$time_stamp);
WordPress Installation receives the request:
$shared_secret = 'foobar';
$timestamp = esc_attr($_GET['timestamp']);
if((time() - $timestamp) > 30) # Threshold is 30 seconds
//do something here - TOKEN expired!
$token = md5($share_secret.$timestamp);
$token_to_check = esc_attr($_GET);
if($token == $token_to_check)
//authenticated!
