Basically, it turns out you need two dedicated IP addresses, or a third vhost.
You could use two IP addresses and do something like this in your apache configuration:
#IP address for WP
#For everything else
Then, make sure the
VirtualHost for WordPress is declared thus:
and all other
VirtualHost blocks declared before WordPress use the other one:
That should work. I haven't tested this AT ALL, so it may not work. Also, I'm not sure using the asterisk in the second
NameVirtualHost line will work; again: untested.
Assuming your domain mapping plugin allows for this, use the 'cname' method of routing mapped domains instead of the 'IP' method. Use something like 'map.domain.com' and have mapped domains point a cname to that (this will prevent them from using the root of their domain, but they should be able to 301 redirect example.com to www.example.com)
Add another VirtualHost entry for 'map.domain.com' as the first vhost entry, mirroring the main vhost entry for the domain, specifically: the document root.
Again, this is also untested.
SUBDOMAIN CONFIGURATION (OLD ANSWER)
I'll use my own site's virtual host file as an example. This is all in one file:
# Virtual host for the subdomain first
# Note that the document root and all other paths are different from the domain's primary virtual host below.
CustomLog /path/to/subdomain/directory/logs/access.log combined
Allow from all
# Virtual host for the main site next
CustomLog /path/to/main/site/directory/logs/access.log combined
Allow from all
That doesn't have to all be in one file; the important part is that Apache loads the more specific virtual host first. That means any virtual hosts with no wildcards in any
ServerAlias values must be loaded before any virtual hosts with those wildcard values.
There is no value, no setting, nothing you could possible do with, in or around WordPress that could possibly solve your problem. By the time the request has reached WordPress, it's too late. Apache has already loaded the virtual hosts, already resolved the hosts, and will always route the traffic the way it does. If the traffic is getting to WordPress when it shouldn't be that is a server configuration issue. Not WordPress.