Basically, it turns out you need two dedicated IP addresses, or a third vhost.
IP Address
You could use two IP addresses and do something like this in your apache configuration:
#IP address for WP
NameVirtualHost 12.34.56.78:80
#For everything else
NameVirtualHost *:80
Then, make sure the VirtualHost
for WordPress is declared thus:
<VirtualHost 12.34.56.78:80>
and all other VirtualHost
blocks declared before WordPress use the other one:
<VirtualHost *:80>
That should work. I haven't tested this AT ALL, so it may not work. Also, I'm not sure using the asterisk in the second NameVirtualHost
line will work; again: untested.
CNAME
Assuming your domain mapping plugin allows for this, use the 'cname' method of routing mapped domains instead of the 'IP' method. Use something like 'map.domain.com' and have mapped domains point a cname to that (this will prevent them from using the root of their domain, but they should be able to 301 redirect example.com to www.example.com)
Add another VirtualHost entry for 'map.domain.com' as the first vhost entry, mirroring the main vhost entry for the domain, specifically: the document root.
Again, this is also untested.
SUBDOMAIN CONFIGURATION (OLD ANSWER)
I'll use my own site's virtual host file as an example. This is all in one file: