Your problem is that you installed it in a subdirectory of WordPress. Both WordPress and Gallery 3 will use virtual URLs in their domain, and with them both acting on the same folders, neither system knows what to do.
So you have
http://blog.url/gallery3/ which is also
The subdomain will point to that folder without any problem ...
http://g3.blog.url/ will redirect to
http://blog.url/gallery3/ ... but
http://g3.blog.url/something will redirect to
Logically, you'll think this will work, but both Gallery 3 and WordPress will try to eat this URL to pass it to the appropriate page. WordPress will send
http://blog.url/index.php/login/ajax because that's where it thinks you're trying to go.
My recommendation would be to not use a subfolder of the WordPress directory for Gallery 3. Put it in a separate folder altogether and point your subdomain there. Then tell WordPress to reference the subdomain rather than the subfolder. You won't run into
.htaccess collisions that way.