I think the best way is to use the before_wp_tiny_mce() hook. Then, you can define the url in PHP; and pass it to the page so it is available to TinyMCE.
function sgp_before_wp_tiny_mce() {
?>
<script type="text/javscript">
var sgp_plugin_url = "<?php echo plugins_url('shortcode_generator_popup.php', __FILE__); ?>";
</script>
<?php
}
add_action('before_wp_tiny_mce', 'sgp_before_wp_tiny_mce');
You may need to change the plugins_url() to match the appropriate location of your php file.
Then, you can use the variable in your plugin tinymce script to get the location.
ed.windowManager.open({
file : sgp_plugin_url,
I haven't tested this... it's from memory. Let me know if you encounter any issues and I'll adjust accordingly.