If you're building something for public consumption (a plugin, a theme, etc) use
admin-ajax.php like you should because that is the appropriate and accepted way to do things and gives your end users the power they need to change and modify things if they so choose.
Beyond that, the best you MAY be able to do is use the
SHORTINIT constant. Define it in a custom php file, then require
wp-load.php and do what you need to do.
SHORTINIT stops most of the WordPress core from being loaded.
// you'll have the basic API here, including `get_option`. Do stuff.
If you do this outside the WP core, you'll have to guess where
wp-load.php may be -- you won't have any
ABSPATH contant to guide you. That's a risky bet unless you're in full control of the system. In other words, if this is a custom, not publicly-released thing, go for it. Otherwise, use