I understand the problem you are having, I was thinking of the same thing.
It is not really a problem tho, it's more a matter of good programming design and have a good separation of front-end and back-end.
Now the way I have managed to solve this is doing the following:
I hope you are calling your AJAX action with
wp_localize_script. We are going to extend this is a little by implementing it this way:
'my_ajax_url' => admin_url( 'admin-ajax.php' ),
'my_frontend_call' => 'something-here',
'my_nonce' => wp_create_nonce( 'my_nonce' )
my_frontend_call, this is the important one we are going to use.
More instructions on how to set up a good ajax call here: link
Now we have this all set up, now we can easily implement the logic in the backend as this:
// Here we check that we are facing the backend.
// This will also be true if we are facing the
// frontend but making an AJAX call.
// As you say bad, also IMO
if( is_admin() )
//Now here we are going to put the magic
// Now these actions are loaded only when `my_frontend_call` is set
// You can add everything here that only get's loaded on a call where
// `my_frontend_call` is set
Now you can mix this up a little with different variables from different locations.
I don't know why this isn't a known recommendation. In my opinion it's ugly when stuff get's loaded when it is not even needed.
NOTE I still am new to WordPress, so maybe this is not a good method, I hope some of the professionals here can comment on this.